How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

Business and Data Overview

Xiaohongshu is a lifestyle community with over 350 million monthly active users. Its app combines community, e‑commerce and commercial services, generating daily logs of several hundred billion events that drive a wide range of data needs.

The platform supports four major data value streams: (1) analytical reporting for executives and self‑service analytics for operations; (2) data products for advertisers, merchants and internal teams; (3) data services such as user‑profile and feature tags for recommendation and search; (4) AI‑driven insights, reports and recommendations.

In 2024 the underlying infrastructure migrated from AWS to Alibaba Cloud, moving 500 PB of data, executing 110 000 tasks with 1 500 participants across more than 40 departments – a record‑breaking effort in the industry.

Data Architecture Evolution

1.0 – ClickHouse‑based ad‑hoc analysis : Offline warehouses produced wide tables that were loaded into ClickHouse for near‑real‑time queries. This reduced query latency from minutes (Spark SQL) to seconds (ClickHouse) but introduced three drawbacks: high cost of ClickHouse clusters, difficult scaling due to compute‑storage coupling, and stale data because Spark + T+1 processing added latency.

2.0 – Lambda architecture with storage separation : ClickHouse’s MergeTree files were synchronized to object storage and local SSDs, extending the queryable time range and lowering storage cost. A Lambda layer merged Flink‑produced real‑time data and Spark‑produced batch data in ClickHouse, achieving day‑level to real‑time insight. Daily, ~6 000 billion events are streamed through Flink into ClickHouse, supporting over 200 internal products with sub‑10‑second response on datasets up to 10 000 trillion rows.

Local joins on user‑level data improve feature‑level analysis.

Materialized views on enumerated log fields capture 70 % of queries, compressing 6 000 billion rows to ~200 billion.

Bloom‑filter indexes on user IDs accelerate specific user‑behavior queries.

These optimizations delivered second‑level analysis for trillion‑scale data and eliminated the need for ad‑hoc data requests from the data‑service team.

3.0 – Lakehouse with incremental compute : To address the growing split between warehouse storage (object store) and ClickHouse files, as well as divergent compute engines (Flink vs Spark) and lack of ETL in ClickHouse, Xiaohongshu adopted a lakehouse built on Iceberg (storage), Flink (real‑time ingestion), Spark (batch jobs) and StarRocks (query engine). DWS wide tables are materialized in StarRocks for fast T+1 analysis, while raw ODS data remains in Iceberg for exploratory queries.

Performance improvements include automatic Z‑Order sorting and intelligent re‑sorting: after analysis of user queries, 80‑90 % of queries now hit Z‑Ordered files, reducing scanned data from >5.5 TB per table to ~600 GB—a ten‑fold gain. Overall query latency (P90) is now around 5 seconds, and query performance is roughly three times that of the previous ClickHouse‑only setup.

General Incremental Compute

The article defines the classic “data‑impossibility triangle” (freshness, cost, performance) and explains how batch, stream and interactive processing each optimize two of the three dimensions. Incremental compute is presented as a fourth paradigm that satisfies all three by adhering to the SPOT standards:

S – a unified full‑data expression supporting all operators.

P – high performance at low cost.

O – openness, allowing multiple engines to consume the same data.

T – tunable configuration without code changes.

In practice, Xiaohongshu’s incremental compute reduced resource consumption to one‑third of the previous baseline, cut component count to one‑third, and lowered development effort to one‑third, while supporting both relational and non‑relational workloads.

Practical Outcomes and Outlook

Key benefits observed after deploying incremental compute include:

Support for unstructured JSON data via Json Flatter, turning JSON strings into columnar storage and improving compression and query speed.

10× faster inverted‑index queries for user‑experiment data.

Unified architecture that simplifies the end‑to‑end data pipeline, reducing development and maintenance overhead.

By 2025, Xiaohongshu plans to further integrate AI, delivering logical data views, automated on‑demand data trimming, and materialized acceleration for AI‑driven analytics. The roadmap aligns with industry trends toward unified stream‑batch processing, lakehouse maturity (especially Iceberg performance), and AI‑centric data services.

For deeper technical details, the speaker references the “Agent‑Oriented Big Data Architecture” whitepaper, which discusses Memory Lake hierarchical memory, multimodal data governance, and unified metadata & lineage built on Gravitino and OpenLineage.