Taobao's Real-Time Lakehouse: Breaking the Impossible Triangle with Apache Fluss

Problem Background and Technical Evolution

Instant retail (flash sales, food delivery) demands extreme real-time capabilities: operations need 30-second funnel refresh, algorithms require minute-level order prediction iteration, and QA needs second-level canary anomaly detection. The legacy stack — Kafka (TT) + Flink + Paimon + StarRocks — handled moderate scale but hit an "impossible triangle" where latency, consistency, and cost could not be optimized simultaneously. Taobao Instant Retail broke this triangle by moving unification to the storage layer with Apache Fluss, while Ele.me took a pragmatic path of near-real-time lakehouse on Paimon + StarRocks, reserving ultra-low-latency streaming storage for later.

Core Concepts and Architecture Positioning

The old architecture stitched Kafka (stream abstraction) and Paimon (batch/lake abstraction) together with numerous Flink ETL jobs as glue. Each glue layer added latency, cost, and consistency risk. Flink was forced to maintain TB-scale state, schedule dozens of data-movement jobs, and absorb dual-stream join memory bloat.

Fluss unifies stream and batch in the storage layer: it converges stream consumption and batch queries into a single distributed storage kernel, eliminating data copying, format conversion, and extra ETL. Taobao built three layers: data sources (order stream, visit stream, product dimension table, inventory/price streams) written to Fluss Log Tables or KV Tables; Fluss storage-compute layer (Delta Join, Partial Update, Auto-Tiering for real-time compute, multi-table joins, wide tables, and lakehouse sync, providing stateless compute to Flink); and business service layer (operations dashboards, algorithmic order prediction, canary monitoring, product-store analysis).

Ele.me's contrasting route: near-real-time lakehouse on Paimon on OSS, higher real-time on TT/SLS, query via StarRocks — a lighter start for teams without second-level SLAs.

Deep Dive into Underlying Principles

4.1 Dual-Stream Join State Bloat → Delta Join

Conversion rate = ordering users / visiting users, requiring real-time join of visit and order streams. Traditional Flink Regular Join caches all unexpired events in local state. During promotions, hundreds of billions of visit logs retained in long windows pushed single-job state to TB scale; RocksDB compaction piled up, checkpoints degraded from minutes to 10–15 minutes and frequently timed out; state I/O saturated disks, attribution latency degraded from seconds to minutes; effective attribution window was capped by local storage capacity.

Fluss's Delta Join offloads both full streams to Fluss KV tables (backed by RocksDB). Order stream writes with PRIMARY KEY (user_id, ds, order_id), visit stream with PRIMARY KEY (user_id, ds); both use join key user_id as bucket.key (strict prefix of primary key). On arrival, either stream triggers a bidirectional prefix lookup on the other KV table. Flink computes bucketId via MurmurHash and routes RPC precisely to the TabletServer holding that bucket — no broadcast or full scan. Flink operators retain only tiny async query queue state, making the join near-stateless.

4.2 Wide-Table Fanout Coupling → Partial Update

Product-store analysis wide table aggregates 5+ behavior streams (exposure, click, visit, add-to-cart, order). Traditional multi-stream join on Kafka produced a new topic per join, creating tightly coupled fanout pipelines. Pain points: updating a single metric required writing full 100+ column rows (I/O waste); adding a field meant modifying multiple jobs; wide-table write latency was dragged by the slowest stream (straggler); intermediate state grew unbounded, and schema/scaling/model updates required stop-and-reprocess.

Fluss's Partial Update (column-level partial update) lets each Flink job aggregate only its own metrics and write to target columns; field-level merging is handled by Fluss server-side RowMerger. This transforms "N upstream streams + M join jobs" tight coupling into "multiple independent writes, server-side column merge".

4.3 Lakehouse Sync Resource Consumption → Streaming-Lakehouse Unification (Auto-Tiering)

Old architecture required a dedicated Flink consumption job per table synced to Paimon; during promotions dozens of "data-movement" jobs competed with core compute. Fluss's Streaming-Lakehouse Unification / Auto-Tiering automatically tiers data from Fluss to Paimon lake, eliminating independent sync jobs; the same data serves second-level ultra-real-time in Fluss and historical/minute-level analysis in Paimon, unified for StarRocks/BI.

Global real-time UV scenario: old stack used Hologres for ID mapping (high-concurrency lookup hotspots, latency seconds) + Flink UDF maintaining TB-scale Bitmap (checkpoint >10 min) + Paimon compaction adding minute-level latency. New design uses Fluss KV for low-latency ID mapping, aggregation table for stateless local Bitmap aggregation, Auto-Tiering for second-level lake sync.

Core Pipelines and Key Flows

Attribution Pipeline

Order and visit streams write to two Fluss KV tables (composite primary key, bucket.key=user_id) → Delta Join operator performs bidirectional prefix lookup (no large state, MurmurHash routing to single TabletServer) → join result written to Fluss wide table → Auto-Tiering auto-syncs to Paimon → StarRocks/real-time BI serves, metrics refresh at second/30-second level.

Global UV Pipeline

Sparse business IDs mapped to dense integer IDs in Fluss KV → Fluss aggregation table performs stateless local Bitmap aggregation → Auto-Tiering seconds-level lake landing.

Real-World Scenarios and Engineering Practices

Key engineering takeaways (parameters must be validated against own scale):

Table design aligns join key with bucket key : set join key (user_id) as bucket.key and make it strict prefix of primary key so prefix lookup hits the correct bucket; partition by ds with auto-partitioning. This mirrors DiDi's "bucket key binds point-query fixed field" principle — let distribution key equal query/join key.

Statelessness as first principle : any scenario requiring Flink to cache large dimension tables or long-window state should first consider offloading to KV table for point lookup, not blindly increasing state backend memory.

Clear SLA-based layering : second-level operations/algorithms/canary monitoring on Fluss; historical replay and minute-level multi-dimensional analysis on Paimon; high-concurrency point queries and OLAP reports on StarRocks. Don't let minute-level Paimon compaction carry second-level monitoring.

Ele.me path as reference : for teams without second-level hard requirements, Paimon on OSS unified near-real-time lakehouse + StarRocks query, reserving TT/SLS for higher real-time, is a lighter start; introduce Fluss ultra-real-time storage only when "dual-stream join state can't hold" signal appears.

Performance, Cost, and Technology Selection

Mechanism direction: Offloading join state to KV point lookup, converting multi-stream join to column-level partial update, turning data-movement jobs into auto-tiering — these three architectural choices deterministically eliminate state bloat, fanout coupling, and duplicate sync.

Project-side self-reported observations (specific to their data scale and hardware; migration value must be verified by own stress tests):

Promotion 100M+ orders caused single-job state to swell to hundreds of GB–TB.

Checkpoints degraded from minutes to 10–15 minutes.

Global UV old-stack Bitmap state led to checkpoint >10 minutes; Hologres lookup latency reached seconds.

Post-refactor metric refresh at second / ~30-second level.

Selection conclusion: Second-level ultra-real-time, dual-stream join, high-concurrency point lookup, multi-column wide table → Fluss-style streaming storage; minute-level and historical, batch analysis → Paimon; high-concurrency OLAP query → StarRocks. When your Flink jobs show TB-scale state, dozens of sync jobs, wide tables dragged by slowest stream, that's the signal to shift from "compute-layer stitching" to "storage-layer unification".

Production Issues and Troubleshooting Methods

Issue 1: Promotion Checkpoint Timeouts, Frequent Job Failures

Follow "phenomenon → metrics → principle → locate → fix": check if state grows unbounded with window/data volume; if dual-stream join caused, switch to Delta Join to offload state to KV table; then examine RocksDB compaction backlog. Prevention: set join key as bucket key, stress-test window upper bounds at scale.

Issue 2: Unstable Wide-Table Update Latency

Principle: multi-stream join dragged by slowest stream. Fix: adopt Partial Update for independent writes and server-side column merge. Prevention: monitor write latency per behavior stream; after decoupling they no longer block each other.

Issue 3: Second-Level Monitoring Data Stale

Principle: mistakenly used Paimon pipeline with minute-level compaction latency for second-level monitoring. Fix: move scenario to Fluss/Auto-Tiering. Prevention: layer by SLA; second-level scenarios must not sit on lake tables.

Issue 4: Lookup Hotspots / Data Skew

Principle: improper bucket key choice causing concentrated access. Fix: choose high-cardinality field consistent with query key as bucket.key; adjust bucket count if needed. Prevention: during DDL review verify bucket.key equals join/point-query key.

Advanced Interview and Architecture Thinking

Q1: Why does Kafka+Flink+Paimon stitching create an "impossible triangle"? Stream and batch separated at storage layer, connected by Flink glue jobs, incurring state, data-movement, and duplicate join costs in latency/cost/consistency.

Q2: Why does Delta Join make dual-stream join stateless? Both streams land in KV tables; arriving side probes the other by join key; Flink no longer caches full events; join key equals bucket key enabling precise point-query routing.

Q3: What wide-table problems does Partial Update solve? Multiple streams write independently, server-side column merge avoids full-row write waste, fanout coupling, and slowest-stream drag.

Q4: How do Fluss and Paimon divide responsibilities? Fluss handles second-level ultra-real-time, point lookup, dual-stream join; Paimon persists history and minute-level analysis; Auto-Tiering bridges automatically.

Q5 (follow-up): Should your team adopt Fluss now? Depends on genuine second-level SLA and large-state pain points; if absent, start lighter with Paimon+StarRocks per Ele.me's gradual path, introduce Fluss when signals appear.

Summary, Knowledge Distillation, and Next Episode

This article uses Taobao Instant Retail as protagonist to dissect how second-level real-time decisions evolved from "Kafka+Flink+Paimon stitched architecture" to "Fluss storage-layer unification": Delta Join cracks dual-stream join state bloat, Partial Update cracks wide-table fanout, Auto-Tiering cracks lakehouse sync cost; contrasted with Ele.me's Paimon on OSS + StarRocks gradual route. We've grasped the impossible triangle's root cause, three capabilities' mechanisms, and Fluss/Paimon/StarRocks layering boundaries.

One-sentence conclusion: When Flink is forced to maintain TB-scale state and dozens of data-movement jobs, the bottleneck is no longer in the compute layer but in "stream and batch stored separately" — the fix is storage-layer unification, letting Flink return to stateless compute.

Next episode (Part 21, Industry Deep Dive 05) enters Connected Vehicles / IoT : based on Alibaba Cloud's "Fluss + DLF + StarRocks Fleet Safety Monitoring" case, examining high-concurrent vehicle reporting, multi-rule real-time compute, and lake-stream unification in proactive fleet safety operations, and comparing differences with instant retail on "device-dimension writes".