How Xiaohongshu’s Data Architecture Evolved for the Big AI Data Era

Xiaohongshu is a lifestyle community app with over 350 million monthly active users, generating daily logs in the order of several hundred billion events. The business relies on four data value streams: analytics, data products, data services, and AI‑driven insights.

1. Data‑platform background

The original platform followed industry‑standard data‑warehouse modeling and comprised self‑built scheduling, operations, asset‑management, governance, and reporting tools. In 2024 the underlying infrastructure migrated from AWS to Alibaba Cloud, moving 500 PB of data with 110 000 tasks and 1 500 participants across 40+ departments.

2. Architecture evolution

1.0 – ClickHouse ad‑hoc analysis : Offline warehouses produced wide tables that were loaded into ClickHouse for near‑real‑time queries, improving response time from minutes (Spark SQL) to seconds (ClickHouse). Drawbacks were high cluster cost, difficult scaling due to compute‑storage coupling, and stale data because of a T+1 Spark batch.

2.0 – Lambda architecture with storage separation : ClickHouse MergeTree files were synced to object storage and local SSDs, extending the queryable time range and reducing storage cost. A Lambda layer merged real‑time Flink streams and batch Spark outputs in ClickHouse, delivering day‑level to real‑time metrics. Daily ingestion reached ~600 billion rows, and performance optimizations such as local joins, materialized views covering 70 % of queries, and Bloom‑filter indexes reduced query latency to the 10‑second range on 10 TB‑scale scans.

3.0 – Lakehouse with generic incremental compute : To address duplicated storage, dual compute engines, and lack of ETL in ClickHouse, a lakehouse was built using Iceberg for storage, Flink for streaming ingestion, Spark for batch jobs, and StarRocks for fast T+1 analysis. Automatic Z‑order sorting and intelligent indexing cut scanned data from >5.5 TB to ~600 GB (≈10× reduction) and achieved 80‑90 % query hit rate on sorted data. Overall query performance improved threefold compared with the prior ClickHouse setup, with P90 latency around 5 seconds.

3. Generic incremental compute

The article defines generic incremental compute as a fourth‑generation data‑processing model that simultaneously targets high performance and low latency, satisfying the Kappa architecture goals while supporting both relational and non‑relational workloads. It introduces the SPOT standards:

S : Full‑stack support for incremental processing across all operators.

P : High performance with low cost.

O : Open – the system must expose data to multiple engines.

T : Tunable – business‑driven adjustments require no code changes.

Applying these standards at Xiaohongshu reduced resource consumption to one‑third, component count to one‑third, and development effort to one‑third compared with the previous Lambda stack.

4. Results and outlook

Incremental compute is now used in community, search, commerce, and advertising scenarios, delivering the same freshness as Spark T+1 with only 1 800 core versus 5 000 core previously, and achieving 1‑2× speedup over Spark for fresh‑per‑5‑minute workloads. Future directions include tighter stream‑batch integration, further query‑performance tuning on Iceberg, and AI‑driven data‑productization on top of the lakehouse.