Inside X’s Open‑Source Recommendation Engine: How the Grok‑Powered Transformer Works

Open‑source release

On the announced date X platform released the source code of its “For You” recommendation engine at https://github.com/xai-org/x-algorithm. The repository is licensed under Apache 2.0 and contains a Rust codebase with supporting Python scripts.

System overview

The feed combines two content sources: internal posts from accounts a user follows (handled by the Thunder module) and external posts retrieved by the Phoenix recall module from a global content pool. Both streams are processed by the Phoenix model, a Grok‑based Transformer adapted for recommendation tasks. The model predicts interaction probabilities (like, reply, retweet) for each candidate and aggregates them into a final relevance score.

Code organization

phoenix/ – Grok model adaptation, recommendation and recall models (e.g., recsys_model.py, recsys_retrieval_model.py) and related test scripts.

home-mixer/ – Rust orchestration layer that performs candidate completion, query enrichment, scoring and filtering.

thunder/ – Rust module responsible for ingesting and deserializing internal (followed‑account) content.

candidate-pipeline/ – Connects content sources to downstream processing and defines the execution graph.

Seven‑step ranking pipeline

User data retrieval : fetch recent interactions, follow lists and preference settings to build a user profile.

Candidate fetching : retrieve internal candidates via Thunder and external candidates via Phoenix.

Content enrichment : attach missing metadata such as text, media, author information, permissions, video length, etc., to each candidate.

Pre‑filtering : drop duplicates, expired posts, self‑posts, content from blocked or muted accounts, and low‑quality items.

Scoring : run four scorers – Phoenix ML scorer (Grok Transformer output), weighted aggregator, author‑diversity reducer, and OON scorer for out‑of‑network content.

Selection : rank candidates by the combined score and keep the top‑K items.

Final validation : perform a last compliance and quality check before pushing the selected items to the user’s feed.

Key design decisions

All hand‑crafted features are removed; the Grok Transformer learns relevance directly from interaction sequences.

Scoring is performed per‑candidate (isolated inference) to avoid cross‑item interference and to enable caching of scores.

Hash‑based vector lookup accelerates both recall and ranking stages.

The model predicts multiple interaction probabilities rather than a single relevance scalar, providing richer signals for ranking.

The candidate‑pipeline framework separates execution, monitoring and business logic, allowing parallelism, graceful error handling and easy addition of new modules.

Technical notes

The system is primarily implemented in Rust with auxiliary Python scripts. It leverages the Grok‑1 Transformer (open‑sourced by xAI) as the backbone for both recall and ranking. The modular pipeline works as follows: Thunder ingests followed‑account posts; Phoenix performs large‑scale recall from a global pool; candidate‑pipeline stitches the two streams; home‑mixer enriches, filters and scores the candidates; finally, a compliance check validates the output before delivery.