How to Choose the Right Distributed ID Generation Strategy for High‑Scale Systems

Introduction

When a distributed system splits large tables (e.g., user or order tables) into multiple databases/tables, a globally unique primary‑key ID is required. The ID must be numeric, trend‑incremental, short, and fast to query.

Distributed ID Generation Schemes

2.1 UUID

Pros: simple code, generated locally, globally unique, easy data migration.

Cons: unordered, cannot guarantee trend‑increment, string storage slows queries, large storage footprint, not human‑readable.

Typical use case: token generation; not suitable when trend‑increment is required.

2.2 MySQL Auto‑Increment Primary Key

Pros: numeric, incremental, high query efficiency, some business readability.

Cons: single‑point failure if MySQL goes down, high load under high concurrency.

2.3 MySQL Multi‑Instance Auto‑Increment (step)

Each instance starts with a different initial value (1,2,3…N) and uses a step equal to the number of instances (e.g., step=4).

Pros: solves the single‑point problem.

Cons: cannot scale after step is fixed; each database still bears heavy load.

Use case: scenarios where data does not need to be expanded.

2.4 Snowflake Algorithm

Generates a 64‑bit positive integer composed of:

1‑bit sign (always 0)

41‑bit timestamp (difference from a custom epoch)

10‑bit machine identifier (e.g., 5‑bit data‑center + 5‑bit machine ID)

12‑bit sequence number (supports 4096 IDs per millisecond per node)

Pros: can produce 4.096 million IDs per second, IDs are trend‑incremental, bit layout is flexible.

Cons: relies on accurate system clocks; clock rollback can cause duplicate IDs.

2.5 Redis Increment Scheme

Uses Redis INCR to generate IDs such as year + day‑of‑year + hour + Redis counter .

Pros: ordered, readable.

Cons: network overhead for each request.

需求：同时10万个请求获取ID
1、并发执行完耗时：9s左右
2、单任务平均耗时：74ms
3、单线程最小耗时：不到1ms
4、单线程最大耗时：4.1s

Performance is acceptable for moderate load, but it does not start from 1 and may need adjustment.

How Leading Companies Design ID Services

3.1 Database Primary‑Key Block Allocation

Instead of requesting a single ID each time, the ID service allocates a block (e.g., step=1000) and returns max_id and step. The client caches the block in JVM and consumes IDs locally until the block is exhausted, then requests a new block.

3.2 Concurrency Control

When multiple services request a block simultaneously, a distributed lock or database row lock ensures only one service updates max_id at a time, preventing duplicate IDs.

3.3 Burst Blocking Problem

In high‑traffic spikes, several services may exhaust their blocks simultaneously, causing a cascade of blocked requests to the ID service.

3.4 Double‑Buffer Scheme

A double‑buffer keeps two ID blocks (buffer1 and buffer2). When buffer1 usage reaches 10 % of its capacity, a background thread pre‑fetches the next block into buffer2. Once buffer1 is empty, the system switches to buffer2 and repeats the process, smoothing spikes and reducing latency.

Conclusion

The presented schemes range from simple UUIDs to sophisticated double‑buffered block allocation. Top‑tier companies often combine block allocation with double‑buffering and dynamic step adjustment to achieve high throughput, low latency, and resilience against database or Redis failures, while also mitigating ID predictability risks.