How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era

Xiaohongshu is a lifestyle community app with over 350 million monthly active users. Its business spans community, e‑commerce and advertising, generating daily log volumes of several hundred billion events. The company classifies data value into four categories: analytics, data products, data services, and AI‑driven insights.

In 2024 the underlying infrastructure was migrated from AWS to Alibaba Cloud, moving roughly 500 PB of data across 110 000 tasks with more than 1 500 participants, setting an industry‑level record for migration complexity.

1.0 Architecture – ClickHouse‑Centric Ad‑hoc Analysis

The initial stack relied on ClickHouse for interactive queries, feeding data from an offline warehouse processed by Spark SQL. This reduced query latency from minutes to seconds but introduced three major drawbacks: high cluster cost due to CPU‑intensive ClickHouse, difficult scaling because of its compute‑storage coupling, and poor data freshness caused by a Spark T+1 pipeline.

2.0 Architecture – Lambda‑Enabled Real‑Time + Batch

Version 2.0 introduced storage separation by syncing ClickHouse MergeTree files to object storage and local SSDs, extending the queryable time range and lowering storage cost. A Lambda layer merged Flink‑produced real‑time data with Spark‑produced batch data inside ClickHouse, delivering day‑level to real‑time insights. Daily ingestion reached ~6 trillion rows, including user‑profile and tag data for joint analysis. Performance optimizations included:

Local joins per user to support feature‑rich analysis.

Materialized views covering 70 % of queries, compressing 6 trillion rows to ~2 trillion.

Bloom‑filter indexes on user IDs for rapid lookup.

These changes enabled sub‑10‑second responses for over 200 internal products without requiring data‑service tickets.

3.0 Architecture – Lakehouse with Incremental Compute

Version 3.0 adopted a Lakehouse model that unifies a data lake (Iceberg) and a data warehouse (StarRocks). Flink handles log ingestion, Iceberg stores raw data, and StarRocks provides fast T+1 analytics on wide tables (dws). To address query‑performance bottlenecks, the team introduced automatic Z‑Order sorting and intelligent index tuning: query logs are analysed, mismatched indexes are identified, and asynchronous rewrite jobs reorder Iceberg files. This reduced scanned data from >5.5 TB per query to ~600 GB (≈10× reduction) and achieved P90 latency around 5 seconds. Compression doubled compared with the original ClickHouse layout, and overall query performance improved threefold while resource consumption fell to one‑third.

General Incremental Compute and SPOT Standards

Incremental compute is presented as the fourth generation of data processing, aiming to satisfy the classic “data impossibility triangle” (freshness, cost, performance). The authors define four SPOT standards:

S – System must support a unified full‑data expression for all operators.

P – System must deliver high performance at low cost.

O – System must be open, allowing multiple engines to consume the same data.

T – System must be tunable without code changes.

Applying these standards, Xiaohongshu’s incremental compute reduced resource cost, component count, and development effort each to roughly one‑third of the previous architecture.

Validation and Practical Benefits

Functional verification showed low migration cost for Spark jobs to incremental pipelines, full data‑accuracy, and flexible freshness‑cost trade‑offs. Performance tests demonstrated 1‑2× speed gains over Spark for 5‑minute freshness, comparable cost to Spark for full‑order tables, and roughly ¼ of the resource cost of traditional Flink for real‑time aggregation.

Additional optimizations included:

JSON Flatter to store semi‑structured data column‑wise, halving compression size and boosting query speed.

Inverted‑index (Date Skipping) that accelerated experiment‑group queries by 10×.

Unified architecture that lowered development and maintenance overhead for algorithm teams, enabling rapid strategy iteration with near‑real‑time data.

Future Outlook

Looking ahead, Xiaohongshu plans to deepen lake‑warehouse integration, improve Iceberg query performance, and embed AI‑driven analysis on top of the unified knowledge base. The roadmap aligns with industry trends toward stream‑batch convergence, mature lakehouse stacks, and AI‑augmented data services.

References: "Agent‑Centric Big Data Architecture" whitepaper (Tencent, Alibaba, Xiaohongshu), Gravitino & OpenLineage for metadata & lineage, Snowflake’s AI/ML pipeline as a comparative example.