MercsDB: Architecture, Storage, Computation, and Optimization of Tencent's MPP Data Warehouse Engine

Background – MercsDB (formerly HermesDB) is Tencent's internally developed MPP data‑warehouse engine that has evolved over nine years through four major versions to meet increasingly demanding real‑time analytics workloads, supporting tables with thousands of columns, billions of rows, and high‑frequency updates.

Storage Architecture – The engine offers two storage modes: local disk for low‑latency, small‑scale workloads (with potential data loss on node failure) and remote storage (HDFS, Ceph, Ozone) for high availability and fault tolerance. Storage choices depend on HA, latency, and hot‑data requirements.

Column Types and Indexes – MercsDB implements several column families: retrieval columns (indexed for point look‑ups), sorting columns (group similar values for faster scans), compression columns (using BitShuffle+LZ4 for high‑cardinality data), and nested columns (Parquet‑like). Indexes include sparse, skip‑list, inverted, point‑lookup, and LBS indexes, each tailored to specific query patterns.

Computation Architecture – Queries enter via JDBC, HTTP, or gRPC, pass through an SQL routing layer that selects either the native engine or the open‑source Presto engine. The native engine performs filter, aggregation, and limit operations directly on indexed column data, while Presto is integrated via a connector to handle full OLAP workloads.

Operator Push‑Down – MercsDB pushes supported operators (filter, agg, limit, order, etc.) down to the storage layer using a stack‑based query generator, simplifying rule management and reducing traversal passes.

Vectorization and Java Vector API – Performance is further boosted by rewriting critical paths with Java Vector API (available from JDK 16) to leverage SIMD instructions, achieving up to ~16× speedup for batch operations. Consistent vector types and avoidance of boxing, object allocation, and unnecessary function calls are emphasized.

Additional Optimizations – Batch‑processing of formerly single‑call functions, sorted index‑to‑string conversion to improve cache locality, and extensive use of vectorized code have yielded noticeable latency reductions.

Benchmark Results – Using the Star Schema Benchmark (SSB) on 200 GB and 2 TB datasets, MercsDB shows significant speedups over its pre‑optimisation version and outperforms ClickHouse in both single‑node and concurrent query scenarios, thanks to its index‑driven execution model.

Real‑World Applications – Deployed in Tencent's core apps (WeChat, QQ, Video, Games) and specific use cases such as WeChat Pay log search and advertising AB testing, where high‑throughput writes, mixed point‑lookup and full‑scan queries, and low‑latency requirements are critical.

Future Plans – Open‑source the engine, enhance cloud integration, continue vectorization work, improve fault‑tolerance and retry mechanisms, and further refine memory management and availability.