How Twitter Scales Redis to 105 TB RAM and 39 M QPS

Since 2010 Yao Yu has been part of Twitter's caching team, and this article is based on her recent talk "Scaling Redis at Twitter". It explains Twitter's extensive use of Redis for Timeline services and the challenges of operating at massive scale.

Timeline service (one data center) – Hybrid List usage:

~40 TB memory allocated 30 M QPS over 6 000 instances

BTtree (one data center) usage:

~65 TB memory allocated 9 M QPS over 4 000 instances

Key technical points highlighted in the talk:

Redis performs well by aggregating unused resources across many servers. Twitter built two custom data models on top of Redis for ideal performance, but legacy code limits rapid feature addition. Analyzing logs locally on each node before network saturation. Fast‑slow path separation: data path is fast, command/control path is slow. Twitter is migrating to a container environment using Mesos, which introduces new challenges for hardware resource limits. Centralized cluster manager monitors the cluster. Because the JVM is slow and C is fast, Twitter rewrote the cache proxy in C/C++.

Why Redis?

1. Redis drives the Timeline service, the most critical component in Twitter's system. 2. Redis replaces Memcache due to network‑bandwidth problems and the long‑common‑prefix problem.

Network‑bandwidth problem

Memcache performs poorly on Timeline fan‑out; Twitter’s read/write pattern is incremental, but Timeline objects are large. When a Tweet is created it is written to the appropriate Timeline; reads load small batches, and scrolling loads additional batches. Home Timeline can contain thousands of entities, making database reads a performance bottleneck. Incremental read‑modify‑write incurs high overhead and often creates network bottlenecks. At >100 k reads/writes per second, objects larger than 1 KB saturate the network.

Long common‑prefix problem (actually two problems)

Flexible data format creates many independent keys, requiring separate requests per attribute.

Using timestamps for metrics causes repeated caching of long common prefixes.

Balancing metrics and flexible schemas benefits from a hierarchical key space.

· Dedicated clusters are configured based on CPU usage; a 1 % CPU slice can sustain >1 k RPS. · Redis shows excellent performance by exposing server capacity rather than current load. · Twitter first deployed Redis for Timeline in 2010, also using it for Ads. · Redis’s disk features are not used; caching and storage are handled by separate teams. · Hot‑key problems are solved with a hierarchical cache that automatically stores hot keys.

Hybrid List

Hybrid List adds predictable memory performance to Redis.

Timeline is a list of Tweet IDs (small integers).

Redis offers ziplist (space‑efficient) and linked‑list (flexible) types; linked‑list pointers add significant overhead.

For memory efficiency, ziplist is the preferred choice.

Ziplist max size is set to the Timeline size, preventing storage of oversized Timelines.

Adding/removing items from a large ziplist is costly due to memmove and realloc operations.

Write latency can be high when large Timelines are modified and memory is fragmented.

Hybrid List is a linked list of ziplists with a byte‑based size threshold for memory efficiency.

Before Hybrid List, the solution was to expire large Timelines quickly, which was costly for reads.

BTree

BTree was added to Redis to support range queries on hierarchical keys.

Hash maps handle secondary fields, but sorted sets can only sort by a single double score, limiting flexibility.

BSD implementation of BTree provides key and range queries with good performance, though memory overhead is high due to pointers.

Cluster Management

Separate clusters are created per purpose; each cluster runs one or more Redis instances. When data exceeds a single instance’s capacity, the key space is sharded across multiple instances.

Cluster management prevents Redis from being overwhelmed and is essential for scaling.

Redis clusters are not trivially idempotent; network failures can cause retries that affect data integrity.

Clusters provide a global manager; unlike Memcache’s client‑side consistent hashing, Redis clusters can detect and repair failing shards.

Twitter experimented with Twemproxy (initially for Memcache) and later added Redis support, building two proxy solutions—one generic and one Timeline‑specific.

Three cluster communication options: state broadcast, proxy routing, or client‑side management when client count exceeds a threshold.

Twitter prefers simple, transparent server designs over heavy server‑side optimizations.

Client‑side changes are hard to roll out; a single change must propagate to ~100 clients, taking years.

Proxy routing separates fast data path from slower command/control path, simplifying performance tuning.

Adding a proxy layer introduces an extra network hop, but latency remains low (≈0.5 ms on Redis servers, ≈10 ms after Finagle).

Proxy failures do not add extra latency; clients can switch to another proxy without awareness of shards.

Twitter servers can maintain >100 000 open connections without failure.

Cache clusters act as look‑aside caches; on node failure, shards are reassigned and data is synchronized.

C++ proxies provide significant performance gains; the proxy layer now uses C/C++.

Data Insights

Cache failures often stem from client misconfiguration or key‑overuse, not the cache itself.

Diagnosing abuse requires detailed metrics and logs to pinpoint problematic keys or shards.

Service‑oriented architecture does not automatically isolate problems; visibility into each component is essential.

Logging each command at 100 k QPS generates ~10 MB/s per server; transmitting this data can consume ~10 % of bandwidth.

Pre‑aggregating logs on the server reduces network overhead; aggregated summaries are sent periodically.

Storm aggregates these logs for storage and visualization, enabling capacity planning.

Operational insight is critical for detecting hot‑key or traffic‑spike issues.

Redis Wishlist

Explicit memory management.

Deployable (Lua) scripts.

Multithreading to simplify cluster management on servers with >100 GB RAM and many CPUs.

Lessons Learned

Scalable services must have predictable capacity as the cluster grows.

Updating many shards can cause global slowdowns if a single shard lags.

Twitter is moving to a container environment with Mesos; resource limits and memory fragmentation must be carefully managed.

Clear visibility into resource usage at runtime is vital; failures can arise from memory limits or GC pauses.

Data‑driven decisions: compare network, CPU, and disk speeds before optimizing.

Lua scripting is not yet production‑ready in Redis; scripts can affect SLA guarantees.

Redis has potential as a high‑performance stream‑processing platform, given its pub‑sub and scripting capabilities.