Challenges and Optimizations of Hive MetaStore at Kuaishou

Kuaishou built a data warehouse on Hive, storing Hive metadata in MySQL; rapid business growth created performance and stability challenges for Hive and its MetaStore.

Challenges include high query performance demand, usability, extensibility, low cost, massive read traffic (>500k queries/day), extensive dynamic partition usage (millions of partitions), and rapid partition growth.

The solution architecture introduces BeaconServer as a stateless Hook service integrated with HiveServer2 to route SQL to efficient engines (Presto, Spark) and provide auditing, rewriting, and error analysis.

Optimization 1 – Read/Write Separation: Read requests are routed to replicas after ensuring GTID synchronization, reducing master QPS by over 70% and improving stability.

Optimization 2 – MetaStore API Improvements: Skip unnecessary partition scans in DESC, upgrade Spark Hive client to reduce redundant get_functions calls, batch large data scans, add partition name index, and enforce string conversion for partition filters, achieving significant latency reductions.

Optimization 3 – Traffic Control: BeaconServer enforces priority‑based throttling during peaks, protecting MetaStore from overload and ensuring high‑priority requests are served.

Optimization 4 – Federation: Routing is implemented at the HMSHandler level, mapping Hive DBs to different RawStore implementations without modifying the persistence layer, enabling horizontal scaling.

These optimizations collectively enhanced query efficiency, stability, and scalability of Hive MetaStore services at Kuaishou.