Design, Optimization, and Future Directions of Alibaba HBase for Large‑Scale Data Storage

HBase, an open‑source NoSQL distributed database based on Google’s BigTable, has been adopted by Alibaba since 2011 and evolved into a core storage product supporting petabyte‑scale workloads with high‑throughput random reads and writes.

The article outlines HBase’s capabilities, its community contributions, and how Alibaba customizes the project to meet internal requirements for flexibility, infrastructure adaptability, service stability, and cost efficiency.

High‑availability is achieved through multi‑data‑center asynchronous replication, with extensive optimizations such as multi‑threaded source log sending, target sink parallelism, hotspot mitigation, online configuration adjustments, and multi‑link routing to avoid bottlenecks and ensure consistent data flow across regions.

For scenarios demanding strong consistency, Alibaba developed synchronous replication that writes logs to both primary and standby clusters in parallel, providing guaranteed data integrity with minimal performance impact.

Cost‑effective redundancy is addressed by reducing replica counts where possible and implementing cross‑cluster partitioned data copy to maintain data safety while lowering storage overhead.

The HExporter system is introduced as a real‑time data pipeline that pushes HBase changes to downstream systems, offering high throughput, accuracy, time‑deterministic delivery, and monitoring capabilities.

Additional performance work includes asynchronous APIs, prefix Bloom filters for Scan operations, HLog compression, and built‑in coprocessor calculations, all contributing to significant throughput and latency improvements.

Future directions focus on mitigating GC pauses, providing SQL access with global secondary indexes, and containerizing HBase deployments for agile operations.