How Baidu Ensures Transaction Data Consistency with Real‑Time and Offline Reconciliation

Baidu's transaction middleware serves a wide range of products such as Mini‑Programs, Map Ride‑Hailing, Baidu Health, Baidu Wenku, and Baidu E‑Commerce, providing payment, order, and settlement capabilities. Rapid growth in order volume and GMV has created a massive data flow that must remain accurate and consistent across more than ten linked systems, including cashiers, order management, marketing, fulfillment, data center, settlement, fund pool, and accounting.

System Components

Cashier Desk : Aggregates payment channels (WeChat, Alipay, UnionPay, Baidu Wallet, etc.) and generates payment and refund records.

Transaction Order : Core driver linking users, merchants, products, inventory, and after‑sale services; manages order lifecycle, pricing, fulfillment, and refunds.

Transaction Marketing : Handles marketing budgets, inventory, and promotional orders.

Transaction Fulfillment : Executes or cancels services based on merchant contracts, producing fulfillment and cancellation records.

Data Center : Consolidates orders, refunds, fulfillment data, and settlement agreements, generating voucher records.

Settlement Center : Settles voucher amounts to merchants according to settlement rules, producing settlement bills.

Fund Pool : Manages merchant balances, transaction flows, and payouts.

Accounting System : Provides a unified export of transaction data for daily, monthly, and yearly financial reconciliation.

The middleware also integrates with external core systems such as Baixin Bank and multiple aggregated payment channels.

Challenges and Root Causes of Data Inconsistency

Key issues include high concurrency leading to idempotency problems, network failures (datacenter, database, middleware), and online code bugs or incomplete integration processes. These inconsistencies can affect user checkout, cause settlement delays, and increase manual effort for financial reconciliation.

Reconciliation Strategies

Two complementary approaches are employed:

Near‑Real‑Time Reconciliation : Listens to binlog changes via the DTS platform, pushes changes to a Business Process (BP) service, matches upstream and downstream records, triggers alerts, and visualizes results through a self‑service reporting platform.

T+1 Offline Reconciliation : Uses big‑data pipelines (Spark, Spark‑SQL, AFS) to extract, transform, and load (ETL) daily transaction data, then performs batch comparison, error correction, and reporting.

Near‑Real‑Time Reconciliation Workflow

Monitor database binlog via DTS and forward messages to BP.

Consume BP messages, abstract upstream/downstream data structures, and store paired metadata.

Periodically check for unmatched pairs, invoke automatic repair APIs, and raise alerts.

Visualize reconciliation results on the Sugar BI platform.

T+1 Offline Reconciliation Workflow

Data Preparation : Sync transaction and payment‑channel logs to the AFS file system using ETL tools (Pingo platform).

Data Comparison : Perform one‑way or two‑way matching; unmatched records are flagged as anomalies.

Balancing : Apply cross‑period balancing to eliminate differences, using up to one year of historical data.

Reporting : Generate visual dashboards showing discrepancy rates, repair outcomes, and statistical summaries.

Error Handling and Alerts

When automatic repair fails after three attempts, the system escalates the issue via email and SMS alerts to the responsible team, requiring manual intervention.

Conclusion

By continuously evolving its real‑time and batch reconciliation systems, Baidu's transaction middleware maintains high data integrity and timeliness, supporting diverse business scenarios and ensuring reliable financial operations for its ecosystem.