How We Boosted Twitter’s Recommendation Engine Reliability from 2‑9 to 3‑9

Preface

Improving the availability of large‑scale online recommendation services is a systems‑engineering problem that must balance business iteration speed, cost constraints, and limited manpower. The Twitter recommendation engine, which powers hot‑feed, video‑post recommendation and other services, experienced frequent instability as traffic grew and material size increased.

Recommendation Engine Architecture

The engine consists of a front‑end, a central controller, a recall module, a ranking module, and a downstream material store.

The controller collects refresh‑time context (user interests, tweet features, read‑status, etc.).

This context is sent to the recall module, which returns candidate tweet IDs. Recall is split into tag‑based recall (rule‑driven) and model‑based recall (vector similarity).

The ranking module enriches each candidate with feature vectors (e.g., via hash) and scores them with a ranking model.

The controller inserts ads, assembles the final list, and returns it to the front‑end.

Training pipelines continuously ingest engine logs and user‑behavior streams to update recall and ranking models in near‑real time. Updated material is streamed to the material store for immediate loading.

Reliability Challenges

Stability

Single‑instance components (e.g., the ranking engine) would crash, run out of memory, or time‑out after a few hours, especially during traffic spikes. The original material store and read‑status storage could no longer satisfy business demand.

Business Support

Refactoring a codebase of tens of thousands of lines while the system remains live and under cost‑utilization pressure made the effort more than a pure technical task.

Factors for Stable Operation

Robust architecture and high‑quality code.

Automation tools (governance, auto‑scaling, exception handling).

Dedicated operations and monitoring manpower.

When any pillar is weak it becomes the bottleneck; strengthening all three enables the system to handle larger traffic safely.

Refactoring Practice

The overhaul was executed in three phases:

Build tooling to stabilize the system before major code changes.

Concentrate manpower on rewriting core components and improving code quality.

Re‑optimize tooling to reduce ongoing maintenance effort.

Material Storage Refactor

The original store used a memory‑mapped external engine that caused slow I/O, memory fragmentation, and limited material scale. A new two‑level index structure was introduced:

The structure consists of four segments:

Header : basic metadata.

First‑level bitmap index : a bit per field indicating presence, enabling sparse‑field support.

Second‑level offset index : stores the offset of each present field, allowing direct access without fragmentation.

Data segment : actual field values (strings stored as compiled bytes, numbers stored directly).

All material objects are kept in a ConcurrentHashMap, allowing updates of tens of thousands of items per second while guaranteeing contiguous memory layout and eliminating fragmentation.

Read‑Status Service Refactor

The legacy 30‑day read‑status solution used four fixed‑size Bloom filters stored in Redis. Problems included:

Uniform filter size caused high false‑positive rates for heavy users and wasted space for light users.

Redis single‑point failure reduced stability.

Business services had to implement their own read‑status checks.

The new design makes read‑status an independent service with a shared SDK. Bloom filter length is now dynamic: a new filter is created only when the previous one reaches a configurable fill‑rate threshold, allowing per‑segment size adaptation and saving >50 % of memory.

A short‑term in‑memory store provides graceful degradation when the primary Redis cluster is unavailable; metadata is read from the primary for the latest filter and from replicas for older filters.

Auto‑Scaling Tooling

An auto‑scaling mechanism monitors incoming QPS and detects rapid increases (from t1 to t2). When the threshold is crossed, scaling actions are triggered so that additional machines become available by t3. The system also supports graceful degradation (e.g., limiting the number of recommendations) while scaling.

Integration of the new material store reduced engine start‑up time from ~20 minutes to ~5 minutes. Dynamic scaling cut latency during peak events by ~25 % and raised the request success rate from <99 % to >99.9 %.

Results

After a three‑month refactor:

Request success rate improved from below 99 % to over 99.9 %.

Average latency decreased by 25 %.

Material capacity grew to support 5‑10 million items per node without additional hardware.

Engine start‑up time became four times faster.

These gains were achieved by combining Twitter’s mature scaling infrastructure with custom material storage, adaptive read‑status, and automated scaling tooling.