How Huolala Built a Scalable Real‑Time Reconciliation Platform for Millions of Daily Transactions

Background

Huolala’s business growth has pushed daily transaction volume beyond one million orders and more than 500 online systems, making traditional batch reconciliation impractical. Frequent data inconsistencies and hidden loss incidents demand a proactive, real‑time solution.

Distributed System Challenges

In a distributed environment, guaranteeing availability and partition tolerance often sacrifices strong consistency, leading to data mismatches that can cause irreversible financial loss.

Upstream/Downstream consistency

Idempotency

Configuration correctness

Document status accuracy

Business rule validity

Any of these issues can trigger severe financial damage, making real‑time detection essential.

Solution Overview

The platform adopts a data‑source‑centric approach: raw source data is continuously compared with reference data, and logical checks determine correctness. Every intermediate data point is treated as “source data” that can be verified.

1.1 Middleware Partition Tolerance

To handle massive real‑time streams (tens to hundreds of thousands of events per second), the system partitions middleware resources and employs dynamic hot‑partitioning to isolate risk.

High‑throughput data pooling reduces third‑party query pressure, while dynamic cluster allocation balances compute resources across reconciliation tasks.

1.2 Dynamic Adaptive Clusters

Compute resources are allocated based on task priority: high‑priority reconciliation jobs receive dedicated resources, and a “egg‑in‑different‑baskets” strategy ensures high availability.

Dynamic cluster assignment with cloud‑resource slicing.

In‑cluster weighted load balancing to keep node utilization even.

Buffer‑based capacity reservation combined with HPA for automatic scaling.

1.3 System Security Guarantees

Rate‑limit protection at each processing stage.

Automatic circuit‑breaker for failing tasks, with replay capability.

Thread‑pool isolation between reconciliation jobs.

Compile‑time syntax validation to block unsafe scripts.

Reconciliation Flow

2.1 Event Ingestion

Supports DB binlog, MQ, and API ingestion, all configurable. Features include sampling, gray‑release, and rate‑limit settings, plus dynamic MQ loading and rebalance strategies.

2.2 Data Verification

Events are routed to specific reconciliation engines, filtered by Groovy scripts, and matched against pooled reference data. The engine supports custom functions, RPC calls, and delayed reconciliation, with security controls such as script blacklists.

2.3 Result Handling

Verification results are tagged, persisted, and can be traced end‑to‑end. Error handling includes automatic retry, hierarchical alerts, replay, and integration with upstream systems for proactive settlement.

Overall Architecture

The design consists of a configuration platform (online config, debugging, cluster management), a data pool (standardized reference data, caching, high‑throughput queries), and the core reconciliation service (event processing, delay handling, result persistence, worker isolation).

Usage Statistics

Connected to 491 upstream sources, peak 80k+ TPS.

124 downstream sources, peak 20k+ TPS.

840+ online scripts, peak 200k+ QPS.

6k+ delayed reconciliation QPS.

160+ custom public functions.

6 dynamically managed partitioned clusters.

Conclusion

After more than two years of development, Huolala’s “算盘” platform delivers sub‑second reconciliation at scale, handling over 100k transactions per second, and provides a robust financial safety net for the company’s entire ecosystem.