How a Snowflake ID Overflow Crashed Our System and What We Learned

Keywords: distributed ID, Snowflake algorithm, Uidgenerator, online issue

Hello, I am Su San.

A recent online incident was caused by a seemingly tiny Snowflake algorithm implementation. This post records the incident, reflects on the work, and reminds readers to check for similar problems in their own projects.

Accident Scene

The incident occurred on 2024‑11‑20 at 09:40. Operations detected an alert, and multiple business groups reported system anomalies.

Urgent log tracing revealed the following error:

Exception details:

com.xxx.uid.exception.UidGenerateException: Timestamp bits is exhausted. Refusing UID generate. Now: 1732112168

Problem Investigation

Developers quickly arrived at the scene. The project uses a custom 19‑digit UID generated by Baidu's UidGenerator based on the Snowflake algorithm.

The relevant code that threw the exception:

/**
 * Get current second
 */
private long getCurrentSecond() {
    long currentSecond = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis());
    if (currentSecond - epochSeconds > bitsAllocator.getMaxDeltaSeconds()) {
        throw new UidGenerateException("Timestamp bits is exhausted. Refusing UID generate. Now: " + currentSecond);
    }
    return currentSecond;
}

The exception means the timestamp portion of the UID exceeded its maximum value.

Tracing the code showed that the timestamp bits were configured too short.

Root Cause Analysis

UidGenerator principle

Reference: https://github.com/baidu/uid-generator/blob/master/README.zh_cn.md

The Snowflake‑based UID consists of four parts:

sign (1 bit) : fixed sign, ensures UID is positive.

delta seconds (28 bits) : seconds elapsed since the base date 2016‑05‑20, supporting about 8.7 years.

worker id (22 bits) : machine identifier, supporting roughly 4.2 million node starts.

sequence (13 bits) : per‑second sequence, supporting up to 8192 IDs per second.

In our project the delta seconds field uses 28 bits, which can represent timestamps up to around 2024‑11‑20, exactly the date of the outage.

To extend the usable period, we can increase the timestamp bits, e.g., to 31 bits.

Reasonableness Analysis

Proposed new bit allocation:

timeBits = 31 : supports up to 2,147,483,647 seconds (~68.5 years), covering until about 2084.

workerBits = 15 : supports 32,768 nodes, sufficient for most systems.

seqBits = 17 : supports up to 131,071 IDs per second, suitable for high‑concurrency scenarios.

Key considerations:

Ensure correct bit‑wise operations for timestamp, worker ID, and sequence.

Maintain time synchronization across nodes.

Plan for future scalability by allowing flexible bit adjustments.

Implementation Measures

Adjust timestamp bits and redeploy the program.

/** Bits allocate */
protected int timeBits = 31; // 28 -> 31
protected int workerBits = 15; // 22 -> 15
protected int seqBits = 17; // 13 -> 17

The new bit configuration is illustrated below:

Roll out updates in batches, prioritizing core services.

Maintain a release list to track affected services.

Verify versions to ensure production correctness.

Monitor basic flows and confirm the issue is resolved.

Report the incident.

Post‑mortem Summary

No Snowflake algorithm is ever completely innocent during an outage.

Developers must understand core framework principles to locate problems quickly.

Reasonable analysis before emergency measures prevents new or lingering issues.

Clear global references, including code, databases, and business logic.

Learn how historical issues are discovered.

Timely detection of similar problems in other projects is essential.