Improving HBase Availability and Reducing Latency Spikes with Replication‑Based Multi‑Path Reads and ZGC

HBase is a low‑cost, distributed LSM‑based database built on HDFS that serves petabyte‑scale data for Didi’s online services, but its strong‑consistency design (choosing C in the CAP theorem) leads to weak availability and noticeable latency spikes during region migrations, splits, or server failures.

To address these pain points, the team first explored replication‑based solutions: the native region replica feature, a master‑slave replication model, and a failover mechanism built on ZooKeeper, all of which still required user‑side read switching and left backup clusters under‑utilized.

They then implemented a client‑side multi‑path read feature inspired by HDFS hedged reads: the client issues a read to the primary cluster and, after a configurable timeout, concurrently issues the same read to a backup cluster, returning the first successful response. Configuration parameters such as hbase.client.hedged.read, hbase.client.hedged.read.timeout, and hbase.zookeeper.quorum.hedged.read control this behavior.

Performance tests (using YCSB, scanning 100 k rows, and measuring max, P99, P999 latencies) showed that multi‑path reads significantly reduced max and P999 latency, effectively smoothing out latency spikes.

Because GC pauses also contributed to latency spikes, the team evaluated Java garbage collectors and adopted ZGC (available from JDK 15) to replace G1. ZGC offers sub‑10 ms pause times, supports very large heaps (8 MB–16 TB), and eliminates stop‑the‑world phases for most operations.

Compiling the Didi‑customized HBase 1.4.8 on JDK 15 required addressing missing classes, adding export flags, and upgrading dependent libraries and Maven plugins.

Benchmarks demonstrated that ZGC reduced scan P99 latency by ~20 % and P999 latency by ~40 % compared with G1, and under heavy write pressure ZGC kept 99.93 % of GC pauses under 10 ms, preventing RegionServer crashes.

The combined use of replication‑based multi‑path reads and ZGC markedly improves HBase’s availability and latency characteristics, making it more suitable for latency‑sensitive online workloads, though challenges such as eventual consistency and limited write‑path improvements remain.