How We Optimized HBase for 80 Billion Daily Logs: Real‑World Tuning Strategies

Background

Datastream writes massive log data to HBase, ingesting about 80 billion rows and 10 TB per day. HBase offers high concurrent write performance, but its complex distributed architecture can cause stability issues that must be addressed promptly.

Historical Situation

When the team inherited the HBase clusters, they observed occasional instability despite widespread adoption by large internet companies. The focus was on the most problematic cluster, which had 17 servers, over 30 tables, 600+ regions, and more than 50 000 QPS, with highly uneven request distribution.

Optimization

Rowkey Design Issues

Phenomenon : Web UI showed severe hotspotting—some regions received zero requests while others handled millions.

Cause : Rowkeys were generated sequentially (e.g., timestamps), concentrating traffic on a few RegionServers.

Solution : Redesign rowkeys to be random (e.g., hash or MD5 prefixes) for uniform distribution.

Recommendation : Use hashed prefixes when constructing rowkeys to ensure even load across regions.

Region Redistribution

Phenomenon : New machines added to the cluster received far fewer regions, causing load imbalance.

Cause : Inconsistent hardware and automatic balancer only considered region count, not table distribution.

Solution : Manually migrate regions from older servers to new ones and split large regions to spread hotspots.

Recommendation : Disable automatic balance during manual redistribution and keep hardware configurations consistent.

HDFS Write Timeout

Phenomenon : Slow HBase logs and HDFS block creation errors.

Cause : Underlying disks were 100% full; although HDFS reserved 100 GB per disk, the system-level quota blocked writes.

Solution : Increase HDFS reserved space above the system reservation (e.g., >100 GB) and run HDFS balance.

Recommendation : Never let disk utilization reach 100%; monitor and adjust reservations proactively.

Network Topology

Phenomenon : Intermittent slow logs despite stable HBase and HDFS layers.

Cause : One server was on a different switch, causing higher latency and affecting HDFS replica writes.

Solution : Relocate the outlier server to the same switch as the rest of the cluster.

Recommendation : For large‑scale distributed systems, keep all nodes on the same switch when possible, and verify switch egress capacity when cross‑switch deployment is required.

JVM Parameter Tuning

Phenomenon : Periodic performance drops and data backlog.

Cause : Full GC pauses on a RegionServer due to insufficient heap size after traffic increase.

Solution : Adjust JVM heap and GC settings to avoid long pauses.

Recommendation : Monitor GC behavior and size the JVM appropriately; avoid full GC in latency‑sensitive services.

Conclusion

After applying the above optimizations, the Datastream HBase environment became stable for months, with consistent performance and no major alerts.