HBase in Practice: Performance Tuning, Monitoring, and Issue Diagnosis

The talk, originally delivered by Alibaba senior technical expert Yu Li at the second China HBase Community MeetUp, is organized into two main parts: performance optimization and monitoring/problem‑resolution.

Performance optimization focuses on I/O tuning, explaining how different storage media (HDD vs. SSD) affect HBase behavior and how to use compaction and flush throttling (available from version 1.1.0) to limit write throughput. It also discusses Per‑CF flush, multi‑WAL configurations for SSDs, and the importance of not setting throttling limits too low, which can cause HFile write delays.

The article then covers disk layout strategies , such as using multiple WALs on machines with many disks, and the configuration keys hbase.wal.provider, hbase.wal.regiongrouping.strategy, and hbase.wal.regiongrouping.numgroups for multi‑WAL setups.

SSD support is described, including write‑path improvements, storage policies (ONE_SSD, ALL_SSD) introduced in 2.0, and the impact of SSDs on both read and write performance.

The guide highlights version‑specific performance issues , advising against certain releases (e.g., 1.0.3, 1.1.3) and recommending upgrades to 1.4.0 or later for better async WAL handling and latency reduction.

Monitoring sections enumerate essential RPC metrics (Server response time, ProcessCall time, QueueCall time, Total Call time), handler saturation, region‑server health, GC pause times, BlockCache/MemStore sizes, and cache hit ratios (data vs. meta). It explains how to interpret these metrics to pinpoint bottlenecks.

The article also details log‑based troubleshooting , emphasizing the need to enable detailed logging (e.g., JIRA HBASE‑16033/HBASE‑16972) for slow requests, differentiate between long processing time and long queue time, and use tools like jstack on region servers.

Client‑side diagnostics are covered, including back‑off policies, batch request logging, and region‑load awareness to avoid overwhelming busy servers.

Finally, a health‑check mechanism is introduced that periodically sends requests with timeouts and alerts on failure rates, with a note that this feature is still in progress and not yet upstreamed.