How We Unified Real‑Time and Batch Financial Risk Features with StarRocks

Background

Financial risk‑control features (e.g., the timestamp of a user’s last credit‑grant submission) are critical for assessing and mitigating risk. Real‑time feature computation in a financial setting is a classic big‑data streaming scenario that demands sub‑second latency and high QPS.

Lambda vs. Kappa Architectures

To handle the 7‑day real‑time window, the team adopted a hybrid approach: Kappa for features within 7 days and Lambda for older data. In Lambda, raw data from a single source is processed by two pipelines—an online stream that writes to a real‑time store and a batch pipeline that archives to Hive. The query layer merges both results. While this improves throughput, it introduces storage redundancy, duplicated development effort (separate stream and batch code), and complex debugging.

Kappa simplifies Lambda by removing the batch branch and using a message queue as the sole storage. Developers maintain only one stream logic, reducing code and operational cost, but the approach is limited by the queue’s storage capacity and slower re‑processing throughput for large back‑fills.

Unified Compute and Storage Vision

The root cause of Lambda’s drawbacks is the separation of stream and batch processing. Industry research points to two unification directions: compute‑unified (single logic runs both batch and streaming) and storage‑unified (eliminate data silos). The goal is a low‑cost solution that supports both real‑time and offline feature development.

Evaluated Alternatives

Data‑Lake Storage – Storing all data in a lake enables both batch and streaming queries via Hive, Spark, or Flink, but incremental writes are not truly real‑time and query latency is high, making it unsuitable for second‑level risk features.

Hologres – Alibaba’s real‑time data‑warehouse offers massive write throughput, low‑latency queries, and PostgreSQL compatibility. It fits the use case technically, but it is not open‑source and has not been adopted internally, limiting feasibility.

StarRocks vs. ClickHouse – Both are high‑performance OLAP engines. After a feature‑by‑feature comparison, StarRocks was chosen for its MySQL protocol support, real‑time ingestion, and mature ecosystem.

StarRocks Validation

Three validation dimensions were tested on a 20 GB, billion‑row credit table:

Data import – batch load completed in 11 minutes; streaming ingestion achieved ~1 second latency.

Query efficiency – MySQL client queries leveraged StarRocks’ vectorized execution and partition pruning.

Concurrency – A 3‑FE + 5‑BE cluster handled up to 1500 QPS before the upstream test platform became the bottleneck; the observed maximum QPS in production is 1109.

The results confirmed that StarRocks meets the latency and throughput requirements for real‑time risk features.

Architecture Design

The new feature pipeline splits data into a batch layer (full historical load) and a stream layer (incremental changes). Both layers write into the same StarRocks table, eliminating the need for separate offline and online tables. The service layer now queries StarRocks directly via a single SQL statement, making the change transparent to downstream applications.

Table Design and Bucketing

StarRocks’ primary‑key model supports delete‑and‑insert updates, ideal for frequent feature refreshes. To accelerate point‑lookup queries (typically by uid or bizId), tables are hash‑bucketed on a single high‑cardinality key, reducing the scanned data fraction to 1/N. Bucket count follows the vendor recommendation of 100 MB–1 GB per bucket to balance metadata overhead and import speed.

Performance Benefits

Using StarRocks reduced feature delivery time by over 80 % compared with the previous batch‑oriented workflow. It also improved data accuracy (single source of truth), added support for table‑join features, delete operations, and a rich set of SQL functions, thereby simplifying feature engineering.

Future Work

Current limitations include reliance on a single StarRocks cluster (limited disaster recovery) and the lack of a unified solution for log‑type data sources, which remain unsuitable for the existing architecture. Ongoing efforts will focus on adding standby clusters, hierarchical feature management, and exploring alternative designs for log‑driven pipelines.

Key Diagrams