How Alibaba’s HiStore Powers the World’s Largest Columnar Database

Alibaba’s e‑commerce platform stores and queries massive historical data using the HiStore column‑store database; on Double‑11 the engine processed more than 6 trillion records and over 5 PB of raw data, making it the world’s largest columnar database.

Project lead Ye Jianlin explains that historical data queries and analytics are query‑intensive, with real‑time inserts and updates being rare, but requiring massive multidimensional and concurrent query loads.

Traditional row‑store databases struggle with such scale and query performance, while HiStore’s low‑cost, high‑compression columnar design offers a complete solution for massive historical data storage and retrieval.

Built on Aliware Middleware to Tackle Global Scale

HiStore’s architecture, based on column‑oriented storage, compression, parallel processing, snapshot concurrency control, and intelligent indexing, delivers superior cost, query, statistical, analytical, and bulk‑load performance. It leverages the Aliware middleware team to meet Alibaba’s massive traffic and stability requirements.

Outstanding OLAP Performance

Compared with commercial products like SAP HANA, HP Vertica, Teradata, and open‑source projects such as InfiniDB, MonetDB, and ClickHouse, HiStore offers rich features: high‑performance multidimensional queries, multi‑core concurrency, DML support, alter table, temporary tables, high availability, heterogeneous data import, fast data load, compression algorithms, and MVCC. In OLAP scenarios it outperforms traditional transactional databases.

Significantly lower hardware cost: column storage and transparent compression achieve average compression ratios >10:1, up to 40:1 in some cases.

Massive data capacity: high‑speed load tools (2 TB/hour) and high compression enable TB‑scale storage and billions of records.

Supports high concurrency and real‑time multidimensional queries, delivering sub‑second retrieval on massive datasets.

MySQL compatibility: fully supports MySQL syntax and protocol, integrating seamlessly with the MySQL ecosystem.

Linear scalability: combined with TDDL/DRDS for linear growth in storage and processing capacity.

Cost‑effective performance: in billion‑record scenarios, storage cost is only one‑third of InfiniDB and load speed is twice as fast.

High Compression + Columnar Storage Cuts EagleEye Hardware Cost by 90%

HiStore’s efficient compression reduced the EagleEye system’s cluster size by 90%, achieving a 20:1 compression ratio and dramatically lowering costs. The security risk control center also sees average compression of 10:1 with millisecond‑level multidimensional aggregation queries.

Real‑Time Multidimensional Queries Excel in Social‑Security Cloud

Since February 2016, HiStore has supported the Ministry of Human Resources and Social Security’s LEAF6 cloud platform, handling hundreds of data shards, complex configurations, and performance tuning for 50 million users and ~800 billion records, delivering excellent query performance for online grouping and multi‑table joins.

Looking ahead, HiStore will continue to deepen its high‑performance, high‑cost‑effectiveness, and high‑availability advantages within Alibaba’s middleware ecosystem, expanding service‑oriented features and enterprise‑grade controls.