Comprehensive Overview of HBase Architecture, Design, and Operations

Technical Background

HBase originated from Google’s three Bigtable papers and implements a distributed column‑oriented NoSQL database.

Design Purpose

It solves real‑time read/write challenges for massive structured data in big‑data ecosystems, compensating for Hadoop’s lack of real‑time storage.

Design Philosophy

Distributed architecture with column‑store storage.

Technical Essence

Concept: Distributed column‑store NoSQL database.

Column storage: underlying files use columnar format.

NoSQL: supports structured and semi‑structured data.

Core Features

Massive tables with billions of rows and millions of columns; distributed memory for real‑time access; spill to HDFS for overflow; multi‑version support per column family.

Cluster Roles

Client

Provides shell, Java API, and Hue/Thrift interfaces for data access.

ZooKeeper

Acts as the master node, handling leader election, storing metadata, and providing HA for the cluster.

HDFS

Stores HFiles and WALs.

HMaster

Manages region assignment, load balancing, metadata updates, and DDL requests.

RegionServer

Handles client read/write requests, manages regions, writes to WAL, maintains MemStore, and performs compaction.

Logical Storage

Namespace, Table, RowKey, ColumnFamily, Column, Value, Version, and Timestamp define the data model.

Column Store

Unlike row‑oriented RDBMS, HBase stores data column‑wise, offering finer granularity and better performance for semi‑structured data.

DDL

1. namespace
   list_namespace
   create_namespace
   drop_namespace
   describe_namespace
   list_namespace_tables

2. ddl (admin only)
   list
   create
   describe/desc
   drop (requires disable)
   disable
   enable

DML

1. dml
   put   (insert, updates are inserts)
   scan  (range or full table scan)
   get   (single rowkey query)
   delete

Hotspot & Data Skew

Hotspots occur when many requests target a single region; data skew is the resulting uneven load. Solutions include proper row‑key design, pre‑splitting regions, and balanced partitioning.

Pre‑splitting

Creates multiple regions at table creation using SPLITS or SPLITS_FILE, improving load balance and read/write efficiency.

RowKey Design Rules

Uniqueness: each rowkey uniquely identifies a row.

Hashing: avoid sequential keys by hashing or reversing fixed prefixes.

Business‑driven: incorporate frequently queried dimensions.

Combination & length limits (≤100 bytes).

Java API

HBaseConfiguration – create config
HBaseAdmin – admin ops (tableExists, disableTable, deleteTable, …)
HTableDescriptor – table schema (addFamily, createTable)
TableName – table identifier
HColumnDescriptor – column‑family settings (setMaxVersions, setBlockCacheEnabled, …)
NamespaceDescriptor – namespace ops
Get, Put, Delete, Result, Cell, Table, ResultScanner – data operations

Read/Write Flow

Writes go to WAL then MemStore; MemStore flushes to HDFS as StoreFiles; compaction merges StoreFiles; splits occur when regions grow too large. Reads check MemStore, then cache, then HDFS.

LSM‑Tree Model

Log‑Structured‑Merge tree handles WAL, in‑memory sorting, flushing, and compaction to maintain ordered on‑disk files.

WAL, Flush, Compaction, Split

WAL ensures durability; Flush writes MemStore to HDFS; Compaction merges files (minor/major); Split divides oversized regions.

Bulk Load

Converts data to HFiles and loads directly into HBase, bypassing WAL for high‑throughput ingestion.

SQL on HBase

Integrations via Hive (MapReduce), Phoenix (secondary indexes), and Sqoop enable SQL‑like access.

Secondary Indexes

Built by mapping query fields to a separate index table; coprocessors (observer, endpoint) automate synchronization.

HBase Optimization

Manual tuning of Flush, Compaction, Split, and column‑family properties (Bloom filter, versions, TTL, block cache, compression) improves performance.

Comparison with RDBMS

HBase offers horizontal scalability, column‑oriented storage, no ACID or joins, suitable for structured and semi‑structured data; RDBMS provides vertical scaling, row‑oriented storage, full ACID, and joins.