Lindorm: High‑Performance Distributed NoSQL Database for Big Data

Lindorm is a cloud‑native, distributed NoSQL database derived from Alibaba's internal HBase branch, offering hybrid storage that combines high throughput, low latency, and multi‑model capabilities for petabyte‑scale workloads.

Performance optimizations such as deep RPC, memory management, and cache improvements enable Lindorm to achieve more than five times the throughput of vanilla HBase, with latency under 3 ms in real‑world double‑eleven traffic, verified by YCSB benchmarks.

Key innovations include a Succinct Trie index that compresses index space up to 12×, ZGC‑based pauseless garbage collection achieving average 5 ms pauses, a lock‑free LindormBlockingQueue that boosts queue throughput fourfold, and a VersionBasedSynchronizer that reduces context‑switch overhead.

Additional engineering advances cover lock‑free MVCC, custom thread‑cache counters for metrics, coroutine‑based handler execution using Dragonwell Wisp 2.0, and an Indexable Delta Encoding that doubles random‑seek performance.

To lower the development barrier, Lindorm provides multiple query models: WideColumn (full HBase API compatibility), TableService (SQL‑like API with rich data types and secondary indexes), FeedStream (high‑throughput queue service), and external full‑text indexing via Solr/ES.

High‑availability mechanisms include a shared‑log consistency protocol with tunable consistency levels, client‑side HA failover, and MTTR optimizations that reduce recovery time by more than tenfold.

Designed for cloud environments, Lindorm runs on ECS + cloud‑disk, supports hot‑cold data separation with OSS storage, and offers a serverless HBase option that auto‑scales resources on demand.

Security and multi‑tenant isolation are provided through a built‑in user/ACL system, group‑based resource isolation, and quota enforcement that limits capacity unit consumption per user, namespace, or table.

Example SQL for the TableService model:

-- Main table and index DDL
create table shop_item_relation (
    shop_id varchar,
    item_id varchar,
    status varchar,
    constraint primary key(shop_id, item_id)
);
create index idx1 on shop_item_relation (item_id) include (ALL);
create index idx2 on shop_item_relation (shop_id, status) include (ALL);
upsert into shop_item_relation values('shop1', 'item1', 'active');
upsert into shop_item_relation values('shop1', 'item2', 'invalid');
select * from shop_item_relation where item_id = 'item2';
select * from shop_item_relation where shop_id = 'shop1' and status = 'invalid';

Overall, Lindorm integrates a suite of performance, scalability, and reliability technologies to provide a next‑generation NoSQL platform for big data, online services, and AI workloads.