Design and Practice of Alibaba Cloud's Billion‑Scale Real‑Time Log Analysis

Alibaba Cloud SLS (Log Service) is a one‑stop observability platform that provides powerful data collection, processing, and delivery capabilities; its data‑ingestion tool ilogtail is fully open‑source. The service offers unified hot and cold storage, intelligent tiering, and rich query and analysis functions such as ad‑hoc and correlation analysis.

The system faces four core problems: (1) ultra‑low query latency for online real‑time analysis (seconds‑level), (2) massive data‑processing scale ranging from millions to hundreds of billions of rows per day, (3) extremely high concurrent query pressure (up to 72 000 concurrent queries and tens of thousands of QPS during peak events like Double 11), and (4) ensuring high availability and tenant isolation in a multi‑tenant cloud environment.

Key design solutions include a three‑part query paradigm (query statement, analysis SQL, time range), an LSM‑Tree‑based storage model with a three‑level hierarchy (project → logstore → shard), time‑based shard partitioning, inverted indexing and column‑store for fast data location and selective loading, and data‑locality where compute nodes share the same machine with storage nodes using domain sockets and shared memory. A layered cache is employed at the index, column, metadata, plan, and result levels, and RPC long‑connections replace short HTTP connections to reduce connection overhead. Storage‑compute separation and horizontal scaling address the massive data‑scale challenge, while affinity scheduling and load‑balancing mitigate hotspot and latency issues.

Future work focuses on further optimizing multi‑tenant resource scheduling, introducing vectorized execution to accelerate computation, and exploring innovative features to enhance user experience.

In the Q&A, the speaker explains how inverted indexes are combined with memory for efficient retrieval, the handling of hot versus cold data latency, and the strategies for block splitting based on size or entry count.