How FeatureKV Handles Billions of Reads and Writes for WeChat Services

Background

WeChat serves over a billion users, generating massive read/write demands for features such as "Look", ads, payments, and mini‑programs. Existing strong‑consistency storage PaxosStore cannot meet the offline‑production, online‑read‑only scenario, leading to two "billion‑level" challenges: 10⁹ QPS reads and 10⁹ writes per hour.

Key Challenges

The "Look" team needs a storage system that decouples model storage from computation, allowing model size to exceed a single machine’s memory.

CTR scoring requires fetching millions of features with the same version; PaxosStore’s BatchGet cannot guarantee version consistency.

Business expects 10⁹ QPS reads; PaxosStore’s replica count is fixed and cannot be increased.

The system must support version and model management, including historical roll‑backs.

FeatureKV Overview

FeatureKV is a high‑performance, easily extensible key‑value store built on WeChat’s proprietary distributed file system (WFS) and metadata service (Chubby). It provides:

Excellent read performance: tens of millions of QPS on in‑memory tables (B70) and millions of QPS on SSD‑backed tables (TS80A).

Excellent write performance: can ingest a billion keys (average value size 400 B) within an hour when the remote file system is fast enough.

Easy scalability: horizontal read‑scale by adding Sect replicas; vertical capacity scale by adding Role instances; write‑scale by adding DataSvr instances.

System Architecture

FeatureKV relies on three external services:

Chubby – stores metadata and coordinates distributed components.

USER_FS – the business‑side distributed file system (WFS/HDFS) where raw input files reside.

FKV_WFS – the internal distributed file system that holds DataSvr‑generated files and supports versioned data.

Key components:

DataSvr : a stateless service that reads USER_FS files, performs routing, sharding, indexing, and writes the resulting files to FKV_WFS. It stores task state in Chubby and can be scaled by adding instances.

KVSvr : a stateful service that polls Chubby for new versions, pulls data from FKV_WFS, loads it locally, and serves read‑only requests. Each KVSvr consists of K Sect s and N Role s; a BatchGet is directed to a single Sect, keeping RPC count low.

Write Process

FeatureKV only supports batch writes, either incremental or full. The final design uses a task‑oriented interface where the business places formatted files on USER_FS and submits a write task to DataSvr. DataSvr streams the input, performs format conversion, routing, and indexing, writes the output to FKV_WFS, and updates Chubby. KVSvr then detects the new version and reloads data.

Key lessons learned:

Directly writing raw files from business code caused version‑upgrade difficulties; a task‑oriented approach avoided this.

Write speed depends heavily on client code quality and machine resources; inefficient parsing (e.g., using std::stringstream for floats) can dominate CPU usage.

Task failures due to DataSvr changes or machine faults required a retry mechanism; the final design uses Chubby‑coordinated state for reliable retries.

Read Performance Optimizations

High‑performance hash tables : MemTable uses a custom read‑only hash table achieving 28 M QPS on 16 threads.

libco AIO : For larger data sets stored on SSD, BlkTable and IdxTable use coroutine‑based asynchronous I/O, reaching >1.5 M QPS on TS80A with four SSDs.

Data packet serialization : Custom binary buffers reduce RPC serialization overhead for large batch sizes.

Data compression : Semi‑precision floating‑point compression (via kimmyzhang’s sage library) reduces bandwidth for dense feature vectors.

Distributed Transaction BatchGet

BatchGet must return a consistent version across all keys. Two approaches were considered:

MVCC (e.g., LevelDB) – complex to support full updates.

COW (copy‑on‑write) – uses double buffers; incremental updates copy the previous version then apply changes, offering higher read performance at the cost of double storage.

FeatureKV adopts COW, guaranteeing that a BatchGet always sees a single version. For multi‑role deployments, a global version is aligned by waiting for all KVSvrs to load the latest version before completing a write task.

Version Rollback

Each table stores a rollback version field. When a rollback is requested, the system merges recent incremental versions into a new full version (checkpoint) in DataSvr, allowing second‑level rollbacks when the target version resides in the double buffer.

Offline Write Process Design

FeatureKV supports three table types:

MemTable : fully in‑memory, unordered hash index, limited by RAM.

IdxTable : in‑memory ordered index with on‑disk values, requires extra index file.

BlkTable : block‑level index in memory, unordered values stored in 4 KB blocks on disk; supports unlimited capacity but requires sorting.

For TB‑scale data, the write pipeline is split into Map (data slicing) and Reduce (sorting) stages, similar to MapReduce. Distributed DataSvr instances perform slicing, then a sorting phase merges slices into ordered files. Two slicing strategies were evaluated; the first (each DataSvr outputs all 2400 slices) was chosen for small‑data scenarios.

With these optimizations, FeatureKV can ingest a billion keys (≈1 TB) in ~71 minutes, a dramatic improvement over the previous 120‑minute baseline.

Operational Experience

FeatureKV is deployed in over 10 modules with 270+ machines, serving services such as Look, Search, Ads, Mini‑Programs, Payments, and user‑profile pipelines. Notable metrics:

Peak read throughput: 1.1 billion features/s (average BatchSize ≈ 3900, peak 3 billion features/s).

Latency: 96.3 % of BatchGet requests complete within 15 ms; 99.6 % within 30 ms.

Reliability: 99.999999 % successful transactional BatchGet.

Business impact includes a 21.8 % increase in ad retrieval volume and a 14.3 % revenue lift for WeChat Ads, and dramatically faster model updates for WeChat Pay.

Conclusion

FeatureKV was created to address the emerging need for massive periodic batch writes combined with online read‑only services, providing strong version management, high scalability, and fault tolerance. Its design demonstrates how a custom key‑value store can meet billion‑level QPS and QPH requirements while maintaining low latency and high availability.