How Twitter Scaled Timelines: Fan‑Out, Redis, and Architecture Secrets

The article examines Twitter's early architectural decisions made to cope with rapid user growth, focusing on timeline generation and fan‑out strategies.

Around 2012‑2013, Twitter served roughly 150 million users, processing about 6,000 tweets per second, which was manageable for writes, but read traffic surged to 300,000 requests per second when users loaded their home timelines.

Fetching a home timeline required aggregating tweets from thousands of followees, which would overwhelm a traditional database.

Get all the tweets ordered according to time, if it belongs to any of these 5000 users
如果某个用户在这5000个用户内，获取该用户所有的推文，并按时间排序

Twitter solved this by maintaining two types of timelines:

User timeline – the collection of a user's own tweets stored on disk.

Home timeline – the aggregated tweets from accounts a user follows, precomputed and stored in a Redis cluster.

By precomputing home timelines and keeping them in Redis, Twitter avoided costly database reads. When a user posts a tweet, the system fan‑outs the tweet to the timeline queues of all followers, creating a write‑heavy but read‑light workload.

The serving flow is:

User requests the home page.

Twitter looks up the user's home timeline in Redis.

The timeline is returned directly to the user.

When a tweet is posted, it is copied into each follower's timeline queue.

Additional details include:

A graph database stores follow relationships, consulted during fan‑out.

Fan‑out is replicated across three machines for fault tolerance.

Only tweet IDs are stored in the cluster; full tweet content is fetched on demand.

Timelines for inactive users (no login for 30 days) are evicted from Redis and rebuilt from disk when needed.

Each home timeline displays a maximum of 800 tweets to control memory usage.

For celebrity accounts with millions of followers (e.g., Lady Gaga with ~31 million fans), Twitter uses a hybrid approach: it skips immediate fan‑out and merges tweets into timelines only when a home timeline request occurs, preventing massive write amplification.

In summary, Twitter’s architecture combines precomputed timelines, Redis caching, selective fan‑out, and replication to achieve high scalability and low latency for massive read workloads.

References:

Timelines at Scale

The Infrastructure Behind Twitter: Scale

Scaling Push Messaging for Millions of Devices @Netflix