How to Scale Systems: From Load Metrics to Architecture Strategies

Load Characterization

Define load with measurable parameters such as requests processed per second, write‑to‑read ratio, concurrent active users, and cache hit rate. Peak values are often more informative than averages for scalability planning. Example from Twitter: posting a tweet averages 4.6 k req/s (peak ~12 k req/s) while home‑timeline reads average 300 k req/s. The primary scalability challenge is the fan‑out caused by each user following many others.

Data Access Strategies

Relational pull model : Store all tweets in a global table. To generate a timeline, join the tweet table with the follows table to retrieve all followees’ tweets and sort by timestamp. Example query:

SELECT tweets.*, users.*
FROM tweets
JOIN users ON tweets.sender_id = users.id
JOIN follows ON follows.followee_id = users.id
WHERE follows.follower_id = current_user;

Push model (data‑pipeline) : Maintain a per‑user cache (inbox). When a tweet is posted, look up its followers and write the tweet into each follower’s cache. Reads become O(1) per user because the timeline is pre‑computed.

Twitter migrated from the pull model to the push model because read load on home timelines grew faster than write load. With an average of 75 followers per user and 4.6 k tweets/s, the push model requires roughly 345 k cache writes per second. For a user with tens of millions of followers, a single tweet could generate tens of millions of writes, challenging a 5‑second write‑completion target.

Performance Characterization

If load increases while CPU, memory, and bandwidth remain constant, response time degrades due to higher service time, network latency, and queuing delays. Maintaining stable performance therefore requires proportionally adding resources.

Latency vs. Response Time

Response time is the client‑observed interval (service time + network latency + queueing). Latency refers only to the server‑side processing time.

Average vs. Percentiles

Average response time can mask tail latency. Percentiles (p50, p95, p99, p999) describe the distribution. For example, p95 = 1.5 s means 95 % of requests finish faster than 1.5 s, while the remaining 5 % may be much slower.

Practical Percentile Monitoring

Collect response times in a sliding window (e.g., 20 minutes) and compute percentiles periodically. A naïve implementation stores all timestamps and sorts each minute; production systems typically use histograms, t‑digest, or HdrHistogram to approximate percentiles with minimal CPU and memory overhead.

Scaling Strategies

Vertical vs. Horizontal Scaling

Vertical scaling upgrades a single powerful machine. Horizontal scaling distributes load across many smaller machines (shared‑nothing architecture). Horizontal scaling is often required for large‑scale services because a single high‑end server becomes expensive and has limited scalability.

Stateless vs. Stateful Services

Stateless services can be replicated easily across nodes, making horizontal scaling straightforward. Stateful services (e.g., databases) are harder to distribute and often remain single‑node until scaling pressures force sharding or replication.

Hybrid Timeline Architecture

Twitter now uses a hybrid approach: most users’ tweets are pushed to follower caches (push model), while “big‑V” users with massive fan‑out still use the relational pull model and are merged at read time. This balances write cost with low read latency.

Design Trade‑offs

Scalable design assumes workload characteristics such as the most frequent operations and typical fan‑out. Incorrect assumptions lead to scaling failures. Early‑stage systems may prioritize rapid feature iteration over exhaustive scalability engineering.

Key Takeaways

Describe load with concrete parameters before planning scaling.

Prefer latency and percentile metrics over simple averages.

Continuously monitor percentiles using efficient data structures (histograms, t‑digest, HdrHistogram).

Select vertical or horizontal scaling based on cost, complexity, and expected growth.

Combine push and pull models when read‑heavy workloads dominate but occasional massive fan‑out exists.