How WeChat Cut Query Latency from Seconds to 100 ms with Druid Optimizations

Background

The WeChat Multi‑Dimensional Monitoring Platform (referred to as "Multi‑Dimensional Monitoring") provides flexible data reporting and real‑time cross‑dimensional analysis. Core concepts are "protocol", "dimension" and "metric"; dimensions describe attributes such as province, city, carrier, error code, while metrics are aggregated values like average latency and report volume.

Optimization Analysis

2.1 User Query Behavior

Analysis of user and internal service query logs revealed that over 99% of requests are time‑series queries, and about 90% target data older than one day. This pattern causes heavy I/O because each time‑series query spans many dimensions and large data ranges.

2.2 Data‑Layer Architecture

The platform uses Apache‑Druid as its storage and query engine. Druid’s architecture includes Master (Overlord, Coordinator), Real‑time nodes (MiddleManager, Peon) and Storage nodes (Historical, DeepStorage, MetaDataStorage, Zookeeper).

2.3 Why Queries Were Slow

Segments store 2‑4 h of data; each Peon writes to an independent segment, leading to many I/O operations for multi‑day queries.

Large time spans cause MiddleManager and Historical nodes to time out and increase Broker memory usage.

Some protocols have >1 million dimension combinations, dramatically degrading query performance.

Optimization Design

3.1 Split Sub‑Queries

Each query is broken into finer‑grained sub‑queries (e.g., a 7‑day request becomes seven 1‑day sub‑queries). Sub‑queries are dispatched to multiple Brokers, reducing load on any single Broker.

3.2 Sub‑Query Cache with Redis

Results of sub‑queries are cached in Redis. A cache entry includes an update_time and a threshold_time (typically 10 min) to determine data freshness. Cache states:

Cache miss : fetch from Druid Broker, then write back to Redis.

Partial hit : fetch missing recent slice from Druid, merge with cached data, and update cache.

Full hit : serve entirely from Redis.

3.3 Dimension‑Combination Sub‑Queries

For dimension‑enumeration queries, data is stored at multiple granularities (day, 4 h, hour) to support cache hits while limiting Redis I/O. Queries are split into N sub‑queries covering different time windows.

Optimization Results

Cache‑hit rate for sub‑queries > 85% (full hit 86%, partial hit 98.8%).

Requests hitting Druid reduced to ~10% of original volume.

Average query latency improved from > 1000 ms to ~140 ms; P95 reduced from > 5000 ms to ~220 ms.

Conclusion

By analyzing query behavior and redesigning the data‑layer architecture—splitting large queries, introducing a Redis‑backed sub‑query cache, and reducing segment size—the WeChat monitoring platform achieved > 85% cache‑hit rates and brought typical query latency down to around 100 ms, dramatically improving user experience while maintaining scalability.