How WeChat’s Security Data Warehouse Powers Billions of Daily Feature Reads

Business Background

WeChat, a national‑level app with over one billion monthly active users, places great emphasis on security. Without sufficient feature data, security policies would be ineffective. The WeChat Security Data Warehouse serves as the central storage for security feature data, handling trillions of read/write requests daily and providing reliable data support for security strategies.

Why a Data Warehouse Is Needed

Before the warehouse, each business team stored computed features in separate KV clusters, leading to fragmented storage, inconsistent interfaces, and poor data quality. This caused difficulties in sharing features, managing them uniformly, and ensuring reliable access. A unified storage solution was required to centralize feature data, simplify access, and guarantee data quality.

Architecture Evolution

1. Storage Selection

Offline KV – suitable for offline features, offers excellent read performance and version management.

Realtime KV – strong consistency, supports online read/write with TTL and MySQL‑like operations.

Other KV – strong consistency but lacks TTL and is not recommended.

2. Unified Storage Interface

The first version introduced a public realtime/offline KV cluster and an access layer that abstracts KV details. Features are identified by a unique <sceneid, columnid> pair, allowing business teams to read/write without caring about underlying KV types.

3. Read/Write Separation and Multi‑IDC Synchronization

Read traffic far exceeds write traffic, so the access layer separates read and write paths to improve performance. Data is replicated across multiple IDC clusters. Offline features are shared via a common file in all IDC KV clusters, while realtime features use a custom MQ to synchronize across IDC.

4. Asynchronous Write and MQ Replacement

Realtime feature writes were moved to an asynchronous MQ, a lightweight, self‑managed queue that replaces the shared distributed queue, preventing interference from other services.

Operational System

An operational module was added to handle feature lifecycle:

Feature Request : Users submit feature requests via a web UI, which are approved through a workflow.

Feature Launch : Approved features are automatically deployed without manual configuration.

Feature Management : Metadata such as business category, type, owner, and tags can be queried and edited.

Feature Analysis : Tracks source data, computation process, data flow, and storage details.

Feature Value Query & Modification : Allows querying and editing feature values through the web UI.

Feature Data Quality Management : Ensures data quality (details in the next section).

Data Quality Assurance

1. Feature Standardization

All new realtime and offline features must follow a specification document. The system validates feature metadata (type, business classification, etc.) and rejects non‑conforming entries. Standardized C++ programming guides are also provided.

2. Empty‑Run System for Offline Features

Offline feature files are first loaded into a standby KV table. An empty‑run process samples live read requests, routes them to a read‑MQ that compares results against the standby table, and calculates a discrepancy rate. If the rate exceeds a threshold, the upload is blocked; otherwise, the file proceeds to production after additional integrity checks.

The empty‑run workflow includes:

Business initiates data upload; the system writes to a backup KV table.

Enable empty‑run; sampled live reads are diverted to the read‑MQ for comparison.

Analyze discrepancy; if acceptable, continue to final validation and production deployment.

Conclusion

By consolidating scattered feature data into a unified warehouse, providing a single access interface, and enforcing strict standardization, WeChat has built a robust foundation for its security business. The added operational system and data‑quality mechanisms further improve efficiency, reliability, and the overall value of security data.