How to Build a Streaming Reconciliation Engine for Billions of Transactions per Hour

Reconciliation is a crucial step in financial security, and in ultra‑large projects it may involve processing hundreds of billions of USD per day. This article explains the basic principles of reconciliation using a simple supermarket example, then describes the two‑stage reconciliation process (funds vs. transaction‑level), and introduces a streaming reconciliation engine that eliminates batch jobs, achieves high concurrency with virtual threads, and scales to handle 30 billion USD per hour (≈27 777 TPS).

01 Basic Principles of Reconciliation

Using a small supermarket scenario, the article shows how daily sales recorded by payment channels (WeChat, Alipay, cash, etc.) are settled the next day after fees, and how reconciliation ensures that the gross amount after fees matches the actual funds received.

02 Reconciliation Process

Two rounds of comparison are required: (1) compare the total receivable amount with the actual settled amount, and (2) match each transaction detail with the counterpart’s records, typically using a unique key.

03 Streaming Reconciliation

To meet real‑time requirements, the streaming approach removes scheduled batch jobs, processes each transaction as an independent stream, and uses parallelism. With an 8‑core 16 GB machine, a single reconciliation should finish within 50 ms; using Java 21 virtual threads, a concurrency of 300–350 allows 27 777 TPS with roughly 93 machines. Database nodes need 16‑core 32 GB servers, with unique‑key queries supporting 3 000+ concurrent operations across 10 databases and 100 tables.

File splitting, parallel parsing, and configurable data distribution rates are required to handle 100 million records (≈15‑25 GB). The engine parses each line into a standard “institution flow”, distributes it to the stateless reconciliation cluster, matches it against internal records, and records the result.

Reconciliation results fall into four categories: not found, duplicate, amount mismatch, and amount match. Results are aggregated in two levels (time‑based and business‑type) and then processed asynchronously for accounting and discrepancy handling.

04 Additional Considerations

Network bandwidth: at least 50 Mbps for 15‑25 GB files.

Flow distribution can use RPC or MQ with load‑aware routing.

Checkpointing is needed to resume parsing after interruptions.