Why Real‑Time vs Batch Data Diverge 5% and Teams Revert to T+1: The Lambda Architecture Dilemma

When daily data volumes reach the hundred‑billion‑record scale, the classic Lambda stack (Flink + Spark + Iceberg/Paimon + ClickHouse) reveals three fatal drawbacks: (1) latency and cost are mutually exclusive because batch recomputation of the full history incurs a T+1 delay while streaming keeps resources constantly allocated; (2) result consistency breaks down as the two pipelines use different code paths and semantics, leading to a 2%‑5% mismatch that invalidates A/B experiments; (3) operational overhead explodes as stream state management becomes exponentially harder and batch back‑fills require lengthy manual scheduling.

To break this deadlock, the article proposes a new paradigm—General Incremental Computation (GIC)—that abandons the binary choice between batch and stream. GIC computes only the data that actually changed, each piece exactly once, by redesigning the whole stack from engine to storage.

Key technical traits of GIC include:

Active merging: instead of waiting for data triggers, the system can be invoked manually or on a timer, tracking every Append/Update/Delete on base tables.

Standard SQL interface: developers write the same full‑batch SQL used in Spark; the engine automatically rewrites it into incremental logic, supporting joins, windows, CTEs and UDF/UDAFs (except nondeterministic functions like random).

Dynamic data model and unified engine: external sources such as Kafka are ingested via Pipe pipelines into a Lakehouse; a CREATE DYNAMIC TABLE with REFRESH EVERY (e.g., 1 min) drives minute‑level incremental refreshes, eliminating long‑running streaming jobs.

Cost‑based optimizer: each refresh selects the cheapest execution plan based on change size, allowing a single scheduling parameter to trade latency for cost.

Kuaishou case study : three workload tiers were evaluated. In a simple GB‑per‑day workload, latency improved from T+15 min to 5 min with incremental cost only 1/20 of the offline baseline. In a medium TB‑per‑day workload, latency dropped from T+1 h to 5 min with cost < 1/3 of offline. In a complex 10 TB multi‑join scenario, latency fell from T+3.5 h to 50 min with comparable cost. Mixed execution kept overall resource usage on par with offline while moving core business to near‑real‑time.

Xiaohongshu case study : the original Lambda stack suffered > 5% data drift and costly wide windows. Migrating the whole pipeline to a Kappa‑style Lakehouse powered by GIC reduced resource consumption from > 5000 cores to ~1800 cores (‑64%), tightened consistency to < 1% deviation, and expanded metric coverage from dozens to hundreds. The migration leveraged a three‑step approach—target high‑pain scenarios, establish automated <1% reconciliation, then gradually double‑run and cut over—assisted by an AI‑driven “incremental SQL” skill.

The article concludes with a repeatable implementation roadmap: prioritize scenarios (P0‑P2) based on latency needs, use minute‑level refresh for log‑driven and A/B metrics, apply partial‑update tables for high‑frequency dimensions, and adopt the three‑step cut‑over with AI assistance to ensure zero‑downtime migration.