How Xiaohongshu Cut Data Architecture Costs by Two‑Thirds with Incremental Computing

1. Xiaohongshu Data Framework Evolution

Xiaohongshu is a lifestyle community app with over 350 million monthly active users, generating billions of logs daily and requiring both real‑time and offline data processing.

The platform originally used a ClickHouse‑based ad‑hoc analysis architecture (1.0), which offered second‑level query speed but suffered from high cost, difficult scaling, and poor data freshness.

High hardware cost due to ClickHouse cluster requirements.

Scaling challenges because ClickHouse combines storage and compute.

Data latency caused by Spark T+1 processing before loading into ClickHouse.

To address these issues, Xiaohongshu introduced a Lambda‑style architecture (2.0) that separated storage and compute, syncing ClickHouse MergeTree files to object storage and SSD, and merging real‑time Flink and batch Spark data in ClickHouse.

Further evolution led to a lakehouse architecture (3.0) integrating Flink for log ingestion, Iceberg for storage, Spark for batch jobs, and StarRocks for fast queries, achieving second‑level analysis over trillions of rows and supporting over 70% user penetration for data‑driven decisions.

2. General Incremental Computing

Incremental computing aims to break the data “impossible triangle” of freshness, cost, and performance by providing a unified data model, configurable pipelines, and performance comparable to the best of batch, streaming, and interactive processing.

The SPOT standards define four requirements for such systems:

S : Support a full‑set of operators in an incremental fashion.

P : Deliver high performance at low cost.

O : Remain open to multiple engines for AI‑driven workloads.

T : Allow flexible tuning without code changes.

In practice, Xiaohongshu’s incremental compute reduced resource usage to one‑third, cut component count by two‑thirds, and lowered development effort to one‑third of the previous pipeline.

3. Practical Benefits and Outlook

Key optimizations include local joins in ClickHouse, materialized views covering 70% of queries, Bloom filter indexes for user‑level analysis, and Z‑Order sorting to shrink scanned data from 5.5 TB to 600 GB, delivering up to ten‑fold speedups.

Future work focuses on further lakehouse enhancements, AI‑driven BI platforms, and expanding incremental computing to more business scenarios.