Mastering Batch Processing: Core Principles, Pitfalls, and Real‑World Cases

Core Principles

Batch processing continuously receives data without immediate computation. A trigger—time‑based, volume‑based, or scheduled—starts a job that reads a large data set, loops through validation, calculation, and database updates, then finishes silently in the background.

Key Features

Non‑real‑time: results appear after seconds, minutes, or hours.

Bulk execution: groups of records are processed together rather than one‑by‑one.

Runs in the background, not consuming front‑end request channels.

Usually scheduled during off‑peak hours (e.g., midnight) to avoid peak‑time load.

Suited for repetitive, standardized computation logic.

Efficiency gains grow with data volume compared to per‑record processing.

Typical Use Cases

Scheduled statistics: daily/monthly reports, financial settlement, commission calculation.

Data synchronization: bulk copy from database A to B.

Data cleansing: fixing dirty or historical data in bulk.

File batch handling: bulk import of Excel files, bulk document generation, bulk export of reports.

Status batch updates: auto‑close expired orders, deactivate expired memberships.

Log aggregation and offline big‑data analysis (e.g., Hadoop offline computation).

Message batch push: sending billing SMSes at night.

Advantages

Off‑peak execution reduces daytime database and server pressure.

Bulk computation minimizes frequent DB connections, delivering far higher performance than per‑record loops.

Automation cuts manual effort and lowers human error.

Centralized logic simplifies maintenance; unified logs aid troubleshooting.

Enables processing of massive historical data that cannot be handled in real time (hundreds of thousands to millions of rows).

Disadvantages (Common Pitfalls)

Data latency prevents use in scenarios requiring immediate feedback.

Job failures can lock tables or exhaust CPU/IO, potentially crashing the whole system.

Partial crashes lead to inconsistent data—some records succeed, others fail.

Concurrent control is hard; duplicate job starts cause duplicate calculations and data.

Large data volumes may cause jobs to run past business hours.

Precise real‑time monitoring is difficult; issues may only be discovered the next morning.

Implementation Tips (High‑Frequency Interview & Practical Points)

Avoid running batch jobs during business peaks; schedule them at night or low‑traffic periods.

Use distributed locks or task‑status flags to prevent duplicate execution.

Implement checkpoint/restart capability so a crash resumes from the failure point instead of restarting from zero.

Take data snapshots or backups before processing to enable rollback on errors.

Never load millions of rows into memory at once; use pagination or sharding (e.g., split 1 000 000 rows into 100 chunks).

Prefer bulk UPDATE / INSERT statements over row‑by‑row updates to reduce I/O.

Log processing status per record; store failed records in a separate “exception list” for manual review.

Set timeout and maximum runtime; automatically terminate and alert if a job hangs.

Add monitoring alerts for job non‑start, timeout, or high failure rates (e.g., SMS or enterprise‑WeChat notifications).

Remember batch processing is not a full transaction; avoid long‑running transactions that could lock the database.

Real‑World Case Studies

Case 1: E‑commerce Order Commission Settlement

Every night at 1:00 AM a batch job processes the previous day’s orders (hundreds of thousands). It filters completed, non‑refunded orders, reads them in 500‑row shards, calculates distributor commissions according to product rules, updates balances in bulk, and writes settlement statements. Failed orders are recorded in an exception table for morning review. The nightly approach avoids real‑time complexity and handles the 7‑day refund window safely.

Case 2: HR Contract Expiry Reminder

A daily 3:00 AM job scans all active employees, selects those whose contracts expire within 30 days, generates reminder records in bulk, and pushes notifications via OA messages and enterprise‑WeChat. An additional batch deactivates contracts that have not been renewed. The delay (up to 24 hours) is acceptable, but the system adds a distributed lock and a “today‑already‑generated” check to prevent duplicate reminders.

Case 3: Bank Nightly Reconciliation

At 00:30 a bank launches an offline batch to reconcile millions of daily transactions with third‑party payment records. It pulls local transaction logs and external statements, compares them in bulk, categorizes results (balanced, missing on one side, missing on the other), archives balanced data, writes discrepancies to a reconciliation error table, and generates a daily report for finance staff. Strict requirements include zero data loss, checkpoint‑based restart, and permanent audit logs. To avoid data loss on server restarts, the job marks processed shards and resumes from the last checkpoint.

Overall, the article provides a comprehensive guide to designing, deploying, and maintaining reliable batch processing systems, highlighting both performance benefits and operational risks.