Designing Trillion‑Scale Counters: From MySQL to a Custom Redis Engine

When a service must handle millions of queries per second, millisecond latency, and zero counting errors, conventional designs quickly break down.

Step 1 – MySQL Single Table

At low traffic the team stored each counter as a row in a MySQL table, with separate columns for likes, comments, and shares. This straightforward approach works only while the data volume stays modest.

Step 2 – Hash Sharding

As the data grew, a single table could no longer sustain the load. The solution was to shard by hashing the ID (modulo operation) rather than by time, which prevents the latest hot content from concentrating on a single shard. This evenly distributes read and write pressure across many databases.

Step 3 – Full Migration to Redis

When the feed began exposing counters for every post, read concurrency exploded. The traditional DB‑plus‑cache double‑write pattern could not guarantee consistency—writes could succeed in MySQL while the cache remained stale. The team therefore abandoned MySQL entirely and moved all counters into Redis memory.

Step 4 – Asynchronous Peak‑Shaving

Sudden hot spikes (e.g., a celebrity post) can saturate Redis. Introducing a message queue allows increments to be aggregated: the consumer merges multiple "+1" operations for the same ID into a single "+N" write, reducing write frequency by an order of magnitude.

Deep Redis Optimizations – Cutting 70% Memory

Native Redis stores a 64‑bit integer as a string, consuming about 30 bytes per counter because of pointers, encoding headers, and structural overhead. Three optimizations were applied:

Key type conversion : Replace string keys with native Long IDs, shrinking each entry to roughly a dozen bytes.

Giant contiguous array : Drop the regular dictionary and use a double‑hash scheme (first‑level h1(key) to locate a slot, second‑level h2(key) probing on collisions). All insert, lookup, and delete operations follow this path, delivering very high efficiency.

Counter packing : Store comment, like, and share counts together under a single key, eliminating three separate keys and saving a large amount of memory.

Hot‑Cold Separation – Cutting Another 60% Cost

Older posts naturally receive fewer views. Keeping all data in memory is prohibitively expensive, while disk‑only storage is too slow. The solution is tiered storage: recent high‑frequency counters stay in RAM, while historical data resides on SSD. When cold data is requested, a dedicated I/O thread asynchronously loads it into an isolated Cold Cache, avoiding blockage of the main processing path.

The combined techniques reduced the required server count by more than half. The Weibo engineering team has validated this architecture in production for six years, confirming its reliability and cost‑effectiveness.

Reusable Template for Counter Systems

1. Assess scale : ≤10 M records – MySQL + cache; 10 M–100 M – direct Redis; >1 B – custom storage engine.

2. Read/write ratio : Read‑heavy workloads favor pure in‑memory solutions; write‑heavy spikes benefit from message‑queue‑based aggregation.

3. Cost calculation : If vanilla Redis cannot handle the load, add hot‑cold separation with SSD, or consider disk‑based engines such as Pika for graceful degradation.

One‑Sentence Takeaway

Effective counter architecture is not designed in a vacuum; it emerges from real traffic pressures, evolving from MySQL to Redis and finally to a deeply customized storage engine.