How Lindorm Cut Costs and Boost Performance for Alibaba’s Massive Data Workloads

Introduction

2022 marks the fifth year that the multi‑model database Lindorm fully supports Alibaba Group’s Double‑11 shopping festival. Over five years the architecture has evolved from a heavily modified HBase‑based 1.0 version to a unified 2.0 version that runs on a single distributed file system and integrates multiple storage engines and data models. Lindorm is now moving toward a cloud‑native, integrated, and tightly fused multi‑model architecture.

Lindorm serves a wide range of internal businesses—Taobao, Tmall, Alipay, Cainiao, Alibaba Cloud, etc.—and currently runs on more than 40,000 nodes with a data volume exceeding 500 PB . During Double‑11 2022 the system handled a daily throughput of 39.1 trillion operations , average response time under 3 ms , with peak read requests of 3.68 × 10⁸ ops/s and write peaks of 2.22 × 10⁸ ops/s , supporting core scenarios such as the command‑center dashboard, mobile messages, and logistics tracking.

With Alibaba Cloud’s “four‑transformations” strategy (cloud‑native, platform‑based, integrated, intelligent), Lindorm must align its development to these goals.

Problems and Challenges

In 2022 the overall economic environment became unstable, pushing the internet industry toward more precise, cost‑effective development. Key demands emerged:

Business cost‑reduction: storage costs often exceed 50 % of total database cost; customers also seek lower‑cost CPU and memory while maintaining performance.

Efficiency improvement: teams want to focus on business logic rather than infrastructure maintenance, seeking cloud‑based solutions that replace complex HBase/Hadoop stacks.

Technical challenges: Lindorm must further lower large‑scale storage cost, remain compatible with open‑source standards to avoid vendor lock‑in, simplify integration, and improve throughput for diverse workloads.

Lindorm Capability Evolution

Cost‑Reduction Improvements

Dictionary Compression for Seamless Savings

Lindorm’s wide‑table engine, built on an LSM‑Tree, extracts data samples, automatically selects optimal compression parameters, and generates a shared dictionary. The feature is transparent to applications—only a table configuration change is needed. It has been widely applied internally and on public cloud, delivering noticeable cost reductions.

Case Studies

Cainiao Logistics achieved >20 % cost reduction while keeping online service capability unchanged; 80 % of tables saw an average storage reduction of >15 %.

Guotai Property Insurance reduced storage cost by 75 % using dictionary compression and hot‑cold separation.

An internet customer migrated its self‑built HBase to Lindorm and saw a 50 % drop in storage space.

HDD + ESSD Heterogeneous Replication

For >100 TB low‑cost storage, Lindorm can run on HDD‑based ECS instances. By mounting ESSD and using a 1 ESSD + 2 HDD replica scheme, Lindorm achieves high performance for hot data while using erasure coding for cold data, reducing redundancy to 1.375×.

Example : A logistics workload generates 800 GB of daily trajectory data (≈300 TB per year). Hot data (first week) is stored with 1 ESSD + 2 HDD replicas; cold data (older than one week) uses erasure coding, dramatically lowering storage cost.

Case Study Lindorm’s wide‑table engine, combined with open‑source APIs, accelerated Grafana Loki queries by leveraging ESSD replicas.

Time‑Series Dedicated Compression

The time‑series engine uses a TSM architecture (LSM‑Tree‑like) with custom compression (Delta‑of‑Delta, Xor, ZigZag, RLE), achieving up to 15:1 compression.

Case Study Alibaba Cloud IoT Platform migrated to Lindorm’s time‑series engine, cutting storage cost by >50 %.

Massive Timeline Performance Boost

Challenges include exploding timelines in container monitoring, sparse log monitoring, and long‑term system/device monitoring. Lindorm addresses them with:

Pre‑down‑sampling to reduce query load for long‑term data.

Multi‑level sharding to contain timeline growth.

Continuous‑query based pre‑aggregation for efficient aggregation across millions of timelines.

Case Study An internet social platform reduced query latency and cut costs by >50 % after migrating from Apache Doris to Lindorm.

Efficiency Improvements

Deep HBase Protocol Compatibility

Lindorm provides an HBase Proxy module that fully supports the native HBase protocol, allowing existing HBase clients and custom integrations to work without code changes while delivering performance comparable to native Lindorm clients.

Case Studies

Internet customers with heavily customized HBase clients continued using their code via the proxy.

A logistics platform migrated HBase from a competitor’s cloud to Alibaba Cloud Lindorm, maintaining cross‑cloud connectivity.

Enhanced SQL Capabilities

Lindorm’s SQL engine now pushes down sorting, grouping, time‑series scans, and down‑sampling to the storage layer, reducing data transferred to the SQL engine and improving query speed. In a group‑wide A+ analytics project, replacing Phoenix + HBase with Lindorm SQL halved cost and boosted performance.

Rich Data Types

Lindorm now natively supports Geo, Blob, and JSON types, enabling efficient storage and querying of spatial trajectories, large binary objects, and semi‑structured data. Example DDL:

CREATE TABLE vehicle_data (
  vehicle_id VARCHAR,
  vtime BIGINT,
  vender VARCHAR,
  vevent JSON,
  vloc GEOMETRY(POINT),
  blf_data BLOB,
  PRIMARY KEY (vehicle_id, vtime)
) WITH (NUMREGIONS='256');

Case Studies

Risk‑control scenarios leveraged JSON and TTL for efficient data management.

Vehicle‑network use cases stored massive trajectories, performed geofence calculations, and handled large media files via Blob.

Elastic Offline Data Import

A new bulk‑load service runs on Lindorm’s elastic compute engine, charging only for active import tasks. This elastic model can scale from 100 CPU × 10 h to 1000 CPU × 1 h, delivering a 10× performance boost at unchanged cost.

Case Study A community search workload reduced offline import cost while achieving high throughput using the elastic bulk‑load service.

Future Outlook

In the coming year Lindorm will further pursue cloud‑native, platform‑based, integrated, and intelligent strategies, refine cross‑engine unification, and advance DB4AI capabilities to provide lower cost, easier usage, and higher development efficiency for both internal and external customers.

Conclusion

Lindorm’s progress relies on customer feedback and the dedication of its engineering team. The product aims to remain the leading big‑data NoSQL solution by continuously lowering costs, improving performance, and simplifying usage.