Mastering Distributed ID Generation: Strategies, Pros, and Code Samples

1. Why Distributed IDs Are Needed

In monolithic systems, primary keys are often generated by auto‑incrementing the database column, which works fine until the system is split into multiple databases and tables. After sharding, using auto‑increment can produce duplicate IDs across different databases, which is unacceptable for business logic.

2. Requirements for Distributed IDs

Global uniqueness: IDs must never repeat.

Trend‑increasing: Ordered primary keys improve write performance in InnoDB tables.

Monotonic increase: Each new ID should be larger than the previous one, supporting versioning and sorting.

Security: IDs should not be perfectly sequential to avoid easy data scraping; irregular increments are preferred.

3. Distributed ID Generation Schemes

UUID

Database auto‑increment

Segment mode

Redis implementation

Snowflake algorithm

Baidu UidGenerator

Meituan Leaf

Didi TinyID

3.1 UUID

UUID (Universally Unique Identifier) is a 128‑bit value usually represented as a 36‑character string (8‑4‑4‑4‑12). It offers very high performance because it is generated locally without network calls.

Advantages: Extremely fast, no external dependencies.

Disadvantages: Large storage size, potential exposure of MAC addresses, and lack of ordering can degrade database performance.

3.2 Database Auto‑Increment

Even in a distributed environment, MySQL auto‑increment can be used by assigning different initial values and step sizes to each machine. For example, with two machines and a step of 2, one generates odd IDs (1,3,5…) and the other even IDs (2,4,6…). However, scaling requires re‑configuring the step and offsets, which becomes cumbersome as the number of machines grows.

Drawbacks: Only trend‑increasing, not strictly monotonic; database load remains high because each ID request hits the DB.

3.3 Segment Mode

This method fetches a range of IDs (e.g., [1,1000]) from the database and serves them from memory until the segment is exhausted, then requests a new segment.

CREATE TABLE id_generator (
  id int NOT NULL,
  max_id bigint NOT NULL COMMENT 'Current max ID',
  step int NOT NULL COMMENT 'Segment length',
  biz_type int NOT NULL COMMENT 'Business type',
  version int NOT NULL COMMENT 'Version (optimistic lock)',
  PRIMARY KEY (id)
);

biz_type: distinguishes business domains.

max_id: current maximum ID.

step: length of the segment.

version: acts like MVCC for optimistic locking.

When a segment is exhausted, the system updates the max_id and version:

UPDATE id_generator SET max_id = #{max_id+step}, version = version + 1 WHERE version = #{version} AND biz_type = XXX;

Advantages: Mature solution (e.g., Baidu UidGenerator, Meituan Leaf).

Disadvantages: Relies on the database.

3.4 Redis Implementation

Redis provides atomic increment commands (INCR, INCRBY) that guarantee uniqueness and ordering. In high‑concurrency scenarios, a Redis cluster is required, and similar segment/step configuration is needed.

Advantages: High performance, strong ordering.

Disadvantages: Introduces a Redis dependency.

3.5 Snowflake Algorithm

Snowflake, originally open‑sourced by Twitter, generates 64‑bit IDs composed of a sign bit, a 41‑bit timestamp, a 10‑bit machine identifier, and a 12‑bit sequence number.

Advantages: Trend‑increasing IDs, no external services, very high throughput.

Disadvantages: Relies on synchronized clocks; clock rollback can cause duplicate IDs.

3.6 Baidu UidGenerator

UidGenerator is a Java implementation based on Snowflake, supporting custom workerId bits and initialization strategies, suitable for Docker and other virtualized environments. It uses a ring buffer and cache‑line padding to achieve up to 6 million QPS per node.

The bit allocation is:

1 sign bit (always 0).

28 bits for delta seconds since 2016‑05‑20 (≈8.7 years).

22 bits for worker ID (≈4.2 million nodes).

13 bits for per‑second sequence (8192 IDs per second).

3.7 Meituan Leaf

Leaf offers two modes: segment (Leaf‑segment) and Snowflake (Leaf‑snowflake). The segment mode improves on the basic segment approach by using a proxy server to batch‑fetch segments, reducing database pressure.

+-------------+--------------+------+-----+-------------------+-----------------------------+
| Field       | Type         | Null | Key | Default           | Extra                       |
+-------------+--------------+------+-----+-------------------+-----------------------------+
| biz_tag     | varchar(128) | NO   | PRI |                   |                             |
| max_id      | bigint(20)   | NO   |     | 1                 |                             |
| step        | int(11)      | NO   |     | NULL              |                             |
| desc        | varchar(256) | YES  |     | NULL              |                             |
| update_time | timestamp    | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-------------+--------------+------+-----+-------------------+-----------------------------+

Leaf‑snowflake follows the same 1+41+10+12 bit layout as Snowflake but obtains worker IDs automatically via Zookeeper sequential nodes.

3.8 Didi TinyID

TinyID is a Java‑based distributed ID system built on the database segment algorithm, extending the Leaf‑segment approach to support multiple databases and a dedicated client library.

Advantages: Easy integration, mature implementation.

Disadvantages: Depends on database stability; requires clustering for high availability.