Design and Implementation of a High‑Performance Distributed Reconciliation System for Large‑Scale Payment Orders

Background: With daily online order volumes exceeding ten million and massive fund flows, manual reconciliation is infeasible, prompting the development of an automated reconciliation system that ensures transaction correctness, improves fund safety, and reduces operational manpower.

System Overview: The reconciliation platform consists of six independent modules—file download, file parsing & push, platform data collection, execution, result statistics, and an intermediate state module. Kafka is used as a message broker to decouple modules, enable retry mechanisms, and support high‑availability state transitions.

File Download Module: Designed with an interface‑based plug‑in pattern to support multiple external channels (e.g., Alipay, WeChat). The core interface public interface BillFetcher { String[] fetch(ReconTaskMessage message, FetcherConsumer consumer) throws IOException; } is implemented by AlipayFetcher , which builds an HTTPS request to Alipay’s API, downloads the compressed CSV bill, and streams it to a local directory. The module relies on Kafka for retry handling, so no additional retry logic is needed.

File Parsing & Push Module: Handles heterogeneous bill formats (CSV, TXT, etc.) using an RDF‑based standardization approach. The BillConverter<T> interface defines boolean match(String channelType, String name) and void convertBill(...) methods. An example Alipay converter matches .csv.zip files, reads the file with FileReader , converts each row to a POJO, and pushes the normalized data to Hive.

Platform Data Collection Module: To avoid overloading transactional databases, platform data is pre‑synchronized to Hive. The DataCollector interface void collect(OutputStream os) throws IOException; abstracts the extraction logic, allowing different Hive tables to be read via plug‑in collectors.

Execution Module: Executes full‑outer‑join Hive SQL statements that combine platform and channel bills, storing the reconciliation results in dedicated Hive tables for downstream analysis.

Result Statistics Module: After execution, the system aggregates discrepancies such as amount mismatches, status mismatches, day‑cut cases, missing records, and extra records. Each type is flagged for manual review on the front‑end, with alerts for critical issues.

Intermediate State Module: Implements a state‑machine driven by Kafka messages. The UpdateReconStatus class updates the reconciliation status, retries up to three times, and sends alerts (e.g., via Lark) when the retry limit is exceeded, ensuring robust fault tolerance.

Version Evolution: The system has undergone three major releases. v1.0 introduced basic automation; v2.0 added dedicated storage directories, exclusive execution queues, and retry mechanisms; v3.0 focuses on “three‑high” goals—high efficiency (Hive‑based data extraction), high accuracy (six‑nine correctness), and high stability (enhanced retry, alerting, and rate‑limiting).

Conclusion: While reconciliation models vary across business domains, the overall architecture remains consistent. The presented design offers a reusable blueprint for building scalable, reliable reconciliation services in large‑scale payment environments.