Financial Risk Control Lakehouse: DiDi's StarRocks & Ant's Paimon Architecture

1. Problem Background & Technical Evolution

Financial risk control is the "touchstone" of real-time lakehouses. It imposes hard constraints: feature queries are often high-concurrency, sub-second point lookups (risk decisions cannot wait); they need both the latest real-time features and long-window cumulative features (e.g., "cumulative applications in the last 30 days"); data accuracy and auditability are non-negotiable because money is involved; and historical lookback may span months. These overlapping constraints make traditional architectures struggle.

The article examines two enterprises solving two key problems in the risk-control pipeline:

DiDi focuses on "unified feature service" — previously real-time and offline features were developed separately (near 7 days via Kappa, long-term via Lambda), causing logic duplication, inconsistent definitions, and slow delivery. They adopted StarRocks to write both streaming and batch data into the same table and expose a unified feature query interface.

Ant Group focuses on "unified storage foundation" — replacing expensive HBase, Elasticsearch, and other acceleration stores with Paimon to solve multi-copy inconsistency, high cost, and slow freshness. One unifies at the feature-service layer (stream-batch unity), the other at the storage layer (convergence); together they represent two typical paths for financial risk-control lakehouses.

2. What You Will Gain

Understand why the four classic routes — Lambda, Kappa, unified compute, unified storage — each fall short in risk-control feature engineering.

See how StarRocks uses "stream-batch same table + primary-key model + bucketed point query" to serve high-concurrency risk-control features.

Learn how Paimon's First-row, Partial-update, and Sort Compact respectively solve long-window deduplication, ultra-fast reconciliation, and historical lookback.

Master reusable table designs (mirror tables, bucket keys) and storage-convergence methods for risk-control scenarios.

Correctly interpret QPS, P99, and resource-reduction numbers within their specific configuration premises.

Answer interview/review questions like "What architecture for risk-control features? Why not query the data lake directly?"

3. Core Concepts & Architecture Positioning

DiDi's risk-control features essentially use a high-performance OLAP engine to ingest both real-time and historical data and expose a unified query to upstream feature services . Per their blog, the data layer splits into batch (historical) and stream (real-time updates) — both write into the same StarRocks table ; the service layer supports StarRocks SQL, transparent to upstream apps. They did not choose a "pure data lake" because risk-control features demand sub-second, high-QPS serving, while data lakes suffer from micro-batch writes, high latency, and low concurrency.

Ant Group's path is "use Paimon as a unified real-time lake storage to converge data previously scattered across HBase, ES, and ODPS" . Per the Meetup sharing, Ant wanted to unify real-time and offline development modes on Paimon, achieving stream-batch integration on both storage and compute. The core pain point: the same data stored in multiple systems, consistency hard to guarantee, expensive storage costs .

Difference: DiDi treats the OLAP engine as an online feature store (query-service heavy), Ant treats the lake format as unified storage (storage-convergence heavy) ; in practice they often combine — lake format for persistence, OLAP engine for serving.

4. Deep Dive into Underlying Principles

4.1 DiDi: Why Four Legacy Routes Are Incomplete, and Why StarRocks Works

Per DiDi's blog, financial risk-control features (e.g., "latest credit submission time") typically require large-scale real-time processing. Traditionally Kappa handles the last ~7 days, Lambda handles long-term.

Lambda runs raw data through both real-time and batch pipelines simultaneously, merging at query time — logic and storage duplicated.

Kappa removes the batch layer, using a single stream processor, lowering dev cost but limited storage and slow recomputation, unsuitable for historical data.

DiDi further evaluated two unification approaches:

Unified compute : One Flink SQL for both stream and batch. In practice, stream still runs on Flink (custom windows), batch on Spark — still separate, high migration cost.

Unified storage : All data in a data lake, accessed via Hive/Spark/Flink. Great for offline, but writes are micro-batch, latency too high, concurrency too low to meet risk-control's sub-second, high-QPS needs .

Conclusion: a high-performance OLAP engine is needed. DiDi evaluated StarRocks vs ClickHouse and chose StarRocks for reasons (project-side comparison):

Strong Join capability (star schema, adapts to dimension changes; ClickHouse relies on wide tables to avoid Joins).

High concurrency (project claims StarRocks reaches 10k+ QPS, ClickHouse only hundreds).

Primary-key model updates fit business needs.

SSB benchmark: ClickHouse response time ~2.2x StarRocks.

Supports second-level import and online elastic scaling.

Table and bucket design are critical for point-query performance.

DiDi splits data into log-type (unstructured, time-appended, no updates/deletes) and business-data-type (structured, full business snapshots, needs updates/deletes). They use StarRocks primary-key tables for business-data " mirror tables " — because ~86% of features come from business data, mirror tables deliver high value; log-type data is unsuitable. For bucketing, risk-control point queries use fixed parameters (uid, bizId), so they use these high-cardinality fields as hash bucket keys : with N buckets, a single query can skip ~(N−1)/N of data. Too many buckets increase metadata overhead, so they followed the 2.3 version guideline of "100MB–1GB per bucket" conservatively, and used only a single field as bucket key to keep point queries flexible.

4.2 Ant Group: Three Scenarios Powered by Paimon Internals

Per the Meetup sharing, Ant uses Paimon to solve three typical problems.

Long-window cumulative deduplication (Five Blessings campaign, half-month to one-month cumulative UV)

Old pipeline used Flink's First-row operator based on state for short-window deduplication (~1 day state), then dim-joined offline T+1 tables for further deduplication. Why not rely entirely on Flink state? (1) Full user state challenges local storage and compute; (2) on error recovery, upstream streams cannot retain the entire campaign window, so state cannot be fully rebuilt. After switching to Paimon, they use First-row Merge Engine to achieve deduplication (once seen, never emitted again) and Changelog to produce increments consumed by downstream Flink streams for accumulation. Internally, this moves "deduplication state" from Flink memory down to LSM lake storage, making state persistent and rebuildable.

Ultra-fast reconciliation (prize distribution, money-related scenarios)

Assurance teams need full multi-table Joins across multiple DBs for reconciliation. Using Paimon's Partial-update (local update) , during sync they incrementally merge multiple DBs into one wide table — new data only updates the changed keys, no full Join, drastically reducing compute; then batch-style queries run against the wide table for reconciliation.

Offline analytical query acceleration (ticket system, months of detail lookback by user ID)

Old pipeline stored multiple acceleration copies in ES, HBase, ODPS. After migration, Flink writes real-time to Paimon partitioned tables, periodic Sort Compact sorts by query key; frontend queries Paimon directly via Flink SQL Gateway; combined with BloomFilter indexes to filter massive data at query time. The sharing also mentions Bucket Hash Join optimizing dimension-table point lookups: shuffle by bucket so each task loads only its bucket, greatly improving cache hit rate (only supports fixed buckets, deduplicated or Lookup Changelog tables, and depends on Flink-side changes).

5. Core Pipelines & Key Flows

DiDi feature query pipeline

Business DB / logs → stream (real-time, second-level) and batch (historical) dual-write into the same StarRocks primary-key table (bucketed by uid/bizId) → at risk decision time, feature service issues fixed-parameter point queries, bucket pruning skips (N−1)/N data, hits prefix/primary-key index → sub-second response.

Ant wide-table incremental merge pipeline (ultra-fast reconciliation)

Multiple business DB binlogs → Flink real-time writes to Paimon wide table (Partial-update merges incrementally by primary key, no full Join) → assurance team runs batch multi-table reconciliation queries on the wide table.

Moving "full Join at query time" to "incremental merge at write time" is the root of performance and cost improvement.

6. Real Business Scenarios & Engineering Practices

Actionable takeaways:

Risk-control feature service (learn from DiDi)

Prefer primary-key model for business "mirror tables", write real-time and historical data into the same table to avoid maintaining two separate definitions; bind bucket keys strictly to the fixed high-cardinality fields used in feature queries (DiDi makes bucket key mandatory in feature-management UI), enforcing bucket pruning at the governance level; do not force unstructured log data into mirror tables — use a separate pipeline.

Storage convergence (learn from Ant)

Long-window deduplication: First-row + Changelog, push deduplication state down to the lake for replayability and rebuildability; multi-source reconciliation / wide tables: Partial-update incremental merge to avoid full Joins; historical detail lookback: partitioned tables + Sort Compact + BloomFilter, organize data layout by query key.

Strong consistency & audit : Financial migrations must run dual-write reconciliation (covering full business and tidal cycles), retain data lineage and reconciliation records; before cutover, confirm primary-key deduplication semantics, late-data handling match the old system.

7. Performance, Cost & Technology Selection

Mechanism-driven direction

Online high-concurrency point queries: OLAP primary-key table + bucket pruning naturally beats micro-batch data lakes; pushing deduplication state / multi-source merge down to LSM lake storage is naturally cheaper and more rebuildable than "in-memory state + query-time full Join"; converging multiple acceleration stores into one naturally reduces inconsistency risk.

Project-side self-reported numbers

DiDi on a 3 FE + 5 BE test cluster, largest credit table 1B+ rows / 20GB+: batch import ~11 minutes, real-time import ~1 second latency; point query prefix index ~20–30ms, secondary index ~70–90ms, no index ~300–400ms; functional test 770k queries P99 ~75ms, avg ~39ms; bucket index peak ~2300 QPS, secondary index ~18, no index ~10; 13-day observed peak ~1109 QPS; feature config efficiency up ~80%, requirement delivery from ~5 days significantly shortened. Ant scenarios per project: long-window deduplication CPU ~60% reduction, memory ~35% reduction, Checkpoint size & duration ~90% reduction, replay reset ~80% reduction; ultra-fast reconciliation ~80% storage saved, ~70% compute saved; daily offline sync from ~19 hours to under 1 hour, historical detail queries ~5–10s. These numbers are bound to specific hardware, data distribution, version, and query templates — only as "some team achieved this scale" reference.

Selection conclusion: Sub-second high-concurrency online feature point queries → use OLAP primary-key tables (not micro-batch lake direct query); long-window deduplication / multi-source wide tables / historical lookback with cost reduction → evaluate LSM lake formats; combination of both is most common.

Root cause may be data layout not sorted by query key, low cache hit rate. Per Ant experience: use Sort Compact + BloomFilter, Bucket Hash Join to boost cache hits; if still insufficient, evaluate whether that point query should keep a dedicated KV store.

Prevention: before replacement, run POC with real query templates, do not substitute average performance for tail-latency acceptance.

8. Advanced Interview & Architecture Thinking

Q1: Why not use data lake directly for risk-control features?

Key points: Data lake micro-batch writes, high latency, low concurrency — cannot meet sub-second high-QPS online point queries; lake better for persistence and offline, online feature service needs OLAP primary-key tables.

Q2: Why did DiDi choose StarRocks over ClickHouse?

Key points: Join/star-schema, high-concurrency QPS, primary-key updates, SSB performance, second-level import & elasticity — comprehensive fit for risk-control point queries, not just raw throughput.

Q3: Why not rely entirely on Flink state for Ant's long-window deduplication?

Key points: Full state pressures local storage/compute; upstream streams cannot retain entire campaign window, errors prevent full rebuild; sinking to LSM lake (First-row + Changelog) enables persistence and replay.

Q4: Why does Partial-update save resources?

Key points: Multi-source data incrementally merged into wide table by primary key, only changed keys updated, avoids query-time full multi-table Join.

Q5 (end-to-end): Given a risk-control platform requiring auditability and high concurrency, how would you build it?

Key points: Lake format for unified storage & long-window features, OLAP primary-key table for online feature service, stream-batch same table for unified definitions, dual-run reconciliation with lineage retention.

9. Summary, Knowledge Extraction & Next Episode

This article uses DiDi and Ant Group financial risk-control cases to dissect two lakehouse paths:

DiDi at the feature-service layer uses StarRocks stream-batch same table + bucketed point query to solve real-time/offline definition fragmentation and high-concurrency point queries.

Ant at the storage layer uses Paimon to converge HBase/ES , leveraging First-row, Partial-update, Sort Compact to solve long-window deduplication, ultra-fast reconciliation, and historical lookback.

Echoing Chapter 2: we now see why the four legacy routes are incomplete, grasp mirror-table/bucket-key design and storage-convergence methods, and clarify the configuration premises behind every performance number.

One-sentence conclusion: The essence of financial risk-control lakehouse is "online point queries to OLAP primary-key tables, long-window state and multi-source merge to LSM lake storage", guarded by dual-run reconciliation to protect the accuracy bottom line where money is involved.

Next episode (Chapter 19, Industry Deep-dive Case 03) enters advertising & marketing : dissecting Alibaba Mama's Flink + Paimon real-time advertising lakehouse (Alibaba Cloud official blog), focusing on advertising demands for upsert deduplication, DWD/DWS layering, Changelog downstream consumption, and algorithm/BI data sharing, plus model-selection differences from risk-control scenarios.