How WeChat’s Ekko Achieves Ultra‑Low‑Latency Model Updates for Billion‑User Recommendations

The 16th Operating Systems Design and Implementation (OSDI) conference, often called the “Oscars” of computer systems, announced its annual best‑paper list, featuring a paper from Tencent’s WeChat team titled “Ekko: A Model‑Low‑Latency Update Solution for Ultra‑Large‑Scale Recommendation Systems.”

OSDI is a premier venue that brings together top researchers from academia and industry to advance operating system technologies, with an acceptance rate of about 19.4% (49 of 253 submissions) this year.

Ekko originates from WeChat’s internal WePS project and addresses the need for rapid model updates in real‑time social scenarios for a user base exceeding one billion. Existing solutions could not keep up with the scale, prompting the development of Ekko.

Key Components of the Ekko Solution

Efficient P2P model‑update transmission service: Coordinates thousands of globally deployed parameter servers to perform real‑time updates, introduces an improved version‑vector algorithm, and implements a log‑free data‑synchronization mechanism that avoids interference from slow or failed machines.

SLO‑aware model‑update scheduler: Prioritizes important gradients on congested networks using freshness‑SLO and quality‑SLO metrics, allowing Ekko to select updates that most affect recommendation quality for prioritized P2P delivery.

Model state manager: Supports disaster recovery for models exceeding 1 TB in size and more than 1,000 instances, enabling rapid detection of performance regressions and incremental rollback in distributed environments within 2.4 seconds.

In comprehensive tests, Ekko demonstrated several orders of magnitude faster model‑update throughput compared with state‑of‑the‑art deep‑learning recommendation systems, achieving up to a 100× improvement over previous best solutions.

The system has been deployed in WeChat for two years, storing hundreds of terabytes of models across thousands of machines and serving over one billion daily users in scenarios such as Channels, “Look‑One,” and Subscription accounts. After full adoption in Channels, global model‑update latency dropped below 2.4 seconds, driving a 40 % increase in daily active users and an 87 % rise in total video plays within six months.

Existing recommendation system update schemes & Ekko architecture

Accelerated effect of Ekko under heterogeneous bandwidth networks

The paper’s results have been recognized by an international top conference, reflecting Tencent’s long‑term investment in computer‑system research. The WeChat team plans to continue advancing foundational theories and key technologies, applying them to diverse business scenarios to improve product experience and drive industry services.