How WeChat Built a Scalable Security Data Warehouse for Billions of Requests

Business Background

WeChat, with over one billion monthly active users, relies on massive security feature data to enforce policies. Without a centralized source, security strategies would lack reliable data. The security data warehouse serves as the core repository, handling trillions of read/write requests daily.

Security Policy Development Process

Policy creation involves three steps: feature data collection, policy implementation, and feedback evaluation. Feature data collection is critical because data quality directly impacts policy effectiveness.

Why a Data Warehouse Is Needed

Before the warehouse, each team stored computed features in separate KV clusters, leading to fragmented storage, inconsistent management, and poor data quality. This fragmentation hindered sharing, caused interface chaos, and reduced system reliability, prompting the development of a unified data warehouse.

Architecture Evolution

1.0 – Unified Storage and Interface

The first version introduced a public real‑time KV and offline KV cluster with an access layer that hides KV details and provides a unified read/write API using a unique <sceneid, columnid> identifier.

2.0 – Read/Write Separation and Multi‑IDC Sync

Read requests far exceed writes, so the access layer splits read and write paths. Data is deployed across multiple IDC sites; offline features are synchronized via shared files, while real‑time features use a proprietary distributed queue to replicate data across IDC.

3.0 – Asynchronous Write and MQ Replacement

To avoid performance impact from synchronous writes, an asynchronous message queue (MQ) replaces the shared distributed queue, providing lightweight, controllable cross‑IDC synchronization for real‑time features.

4.0 – Operations System

The operations module adds feature request, launch, management, analysis, value query/modification, and data‑quality management functions, eliminating manual configuration edits and enabling automated approvals and deployments.

Storage Selection

Two KV types are used:

Offline KV : Optimized for batch‑computed features, offers high read performance, version control, and stores protobuf objects.

Real‑time KV : Supports low‑latency reads/writes, data expiration, and presents a MySQL‑like table schema where each feature maps to a column.

Data Quality Assurance

Feature Standardization

All new features must follow a strict specification document. The system validates meta‑information (type, business classification, owner, tags) and rejects non‑conforming entries. C++ programming guidelines and examples are provided to ensure consistent implementation.

Empty‑Run System

Offline feature files are checked by an empty‑run pipeline before going live. The pipeline samples live read traffic, routes it through a read‑MQ that compares results against the empty‑run KV, calculates a difference rate, and blocks deployment if the rate exceeds a threshold. Successful checks proceed to final deployment, with alerts for any step failures.

Conclusion

By consolidating scattered feature data, providing a unified access layer, standardizing feature definitions, and implementing robust quality‑control mechanisms, the security data warehouse underpins WeChat's large‑scale security policy development, dramatically improving efficiency, reliability, and data value.