Points System Architecture: Concurrency, Batch Expiration & Idempotent Reconciliation

01 Define System Promises Before Technical Solutions

The article opens with a realistic scenario: a platform with 10 million registered users, where a user holds 110 points split across two batches — Batch B001 (50 points, expires Sep 30) and Batch B002 (60 points, expires Dec 31). The user attempts to redeem 80 points; the request times out, the client retries, and days later the order is refunded. A simple balance column cannot answer: did the first request succeed? Will the retry double-deduct? Which batches were consumed? How should refunds restore points? How to prove final correctness after cache, message, or expiration task failures?

Before designing, three things must be clarified: business actions the system must support, traffic patterns, and inviolable rules (invariants). The full points lifecycle includes issuance, consumption, freeze/unfreeze, refund/adjustment, expiration, query, and reconciliation. Each action demands different data; a balance-only model will collapse when batch expiration and refunds arrive.

External interfaces must carry business party, business order number, and operation type. Returning "points credited" vs "request accepted" implies different commitments and downstream handling.

"High Concurrency" Must Become Verifiable Numbers

Teaching assumptions: 10M users, peak 20k QPS for balance queries, 3k TPS for point changes, reads far exceed writes. These are design inputs, not capacity proofs. Crucially, per-account peak write concurrency matters: 3k requests spread across accounts parallelize easily, but 100 concurrent writes to one account contend on the same row, dictating choice of row locks, optimistic locking, or conditional updates, and whether hot accounts need dedicated rate limiting.

Rules That Must Never Break (Invariants)

Same business operation takes effect exactly once.

Available points never go negative.

Every balance change leaves an immutable history record.

Refunds and corrections create new records, never silently overwrite past ones.

Account balance must be fully explainable by transactions, ledgers, and batches.

Leaderboards and reports can tolerate seconds of delay; balances, idempotency results, and audit history cannot.

02 Overall Architecture: Synchronous Core, Asynchronous Periphery

Balance queries go through API gateway → query service → Redis (cache-aside); on miss, query service falls back to authoritative DB and rebuilds cache. Detail/report queries hit read replicas or dedicated read models. Core mutations (deduct, freeze) always route to account core and authoritative DB.

Key boundary: one local transaction commits account snapshot, transaction, ledger, batch consumption, and outbox events. Notifications, reports, leaderboards update asynchronously via events. The diagram shows business boundaries (access, rules, account, expiration, reconciliation, query) — not necessarily separate microservices from day one. A modular monolith with clear data ownership and dependency direction is fine initially. The account core, responsible for atomic account updates, should not be split too finely, and no other service must bypass its API to touch the account database directly.

03 Data Model: Beyond Balance

A table with only user_id and balance answers "how many now" but cannot explain origin, reason for decrease, or expiry. The following entities are needed:

Account Snapshot — current available/frozen points; updated in same transaction as account changes; serves fast reads.

Points Transaction — records each business operation and its processing state; idempotency key unique; state traceable.

Account Ledger — explains why balance changed, before/after values; append-only, never overwritten.

Points Batch — source, expiry, remaining quantity; supports expiration rules and consumption order (earliest expiry first).

Batch Consumption Detail — links a consumption transaction to specific batches consumed; enables traceability for refunds.

Outbox Event — records changes needing downstream notification; committed in same transaction as core changes.

These entities are not one-to-one: one transaction may generate multiple ledger entries and consume multiple batches; outbox events are for notification, not accounting.

After consuming 80 points, the system produces:

Points Transaction T001: consume 80, success
Account Ledger L001: available -80, balance 110 → 30
Batch Consumption A001: from B001 consume 50
Batch Consumption A002: from B002 consume 30
Account Snapshot: available 30
Outbox Event E001: points.consumed, awaiting downstream notification

The transaction records overall result, ledger explains total balance change, batch consumption details show exact source. This linkage enables refunds to restore original batches and expiries. Snapshot speeds queries, ledger explains results, transaction preserves state, batches manage source/expiry. All update in one local transaction to avoid partial updates.

If points involve multiple issuers, merchant settlement, or inter-account value transfer, double-entry bookkeeping with corresponding entries can track flow. But for a single platform with simple rules, "snapshot + immutable ledger + batches" usually suffices. The test: can balance be fully explained by history?

04 End-to-End Safe Consumption Flow

Step 1: Idempotency Key Confirms Prior Processing

Client timeout ≠ server failure. The core must identify the same operation via a stable business identity:

Idempotency Key = Business Party + Business Order Number + Operation Type

"Consume", "Freeze", "Confirm Deduct", "Cancel Freeze" are distinct operations; they must not share a vague key. Gateway can filter obvious duplicates, but final idempotency result must be persisted in the points core, guarded by a database unique constraint.

Step 2: Complete Accounting in One Local Transaction

For immediate consumption of 80 points, a single transaction does:

1. Create points transaction by idempotency key; if exists, return original result
2. Lock account (or use version for optimistic concurrency)
3. Atomically deduct available balance, ensuring balance ≥ 0
4. Consume batches in "earliest expiry first" order: 50 from B001, 30 from B002
5. Write two batch consumption details and append account ledger
6. Update account snapshot, write pending outbox event
7. Commit transaction and return result

Row lock, optimistic lock, or conditional update — choice depends on per-account conflict probability and load test results. Never read balance, decide in application, then unconditionally overwrite; final deduction must be guaranteed by database concurrency control.

Immediate redemption can deduct directly. Only when an order may later be confirmed or cancelled is a "freeze → confirm deduct / cancel unfreeze" state machine needed. Freeze is not a mandatory step for all consumption scenarios.

Step 3: Update Cache After Database Commit

Using cache-aside (Redis), commit DB transaction first, then delete cache. Next query misses cache, reads fresh value from DB, rebuilds. DB commit and Redis delete are not atomic; delete may fail. Solution: write cache invalidation event in the same account transaction; a background task retries deletion. If the same event must notify multiple systems or needs buffering and independent scaling, a publisher pushes events to a message queue (MQ). MQ is optional for transactional outbox; the outbox table itself can serve as a reliable task queue at moderate scale.

Cache needs reasonable TTL and version. On hot cache miss, use request coalescing or short-term lease to allow only one thread to backfill; re-validate version/lease before writing to prevent a slow request with stale data from overwriting after cache deletion. Regardless, points deduction must rely on DB, never on Redis balance.

Step 4: Cross-System Convergence via Events

Order completion triggers points issuance across order and points systems. Common pattern: order system saves order state and outbox event in one local transaction; background task or CDC publishes committed events; points system consumes event and records points idempotently using business order number and operation type.

Transactional outbox solves "business committed but event not saved" gap, but does not guarantee exactly-once delivery. Senders may retry, messages may delay or reorder; consumers must still be idempotent and handle retries, dead letters, ordering, and reconciliation.

05 Recovering from Failures: Timeout, Refund, Expiration, Reconciliation

Request Timeout: Retry Returns Original Result

The initial consumption request timed out, but with the same idempotency key the core finds transaction T001 and returns its original status, not re-deduct. If first attempt is still processing, return "processing" or let caller poll after a short wait.

Order Refund: Create Reversal, Never Modify Original

Refund creates a new transaction linked to T001 and a reverse ledger entry, not deleting the original ledger. System uses A001/A002 to locate the two batches consumed, then restores points per business rules. If original batches have expired by refund time — whether points expire immediately, retain original expiry, or get new expiry — is a product rule that must be predefined. The technical system preserves the original allocation and records the chosen rule and outcome.

Points Expiration: Process by Batch, Ensure Task Idempotency

Expiration job must not just adjust total balance. It finds expired batches with remaining quantity, generates a stable expiration task ID, and in one transaction reduces batch remainder, updates account snapshot, and appends expiration ledger. For large data, split by expiry date and account range across multiple nodes. Scheduling leases handle work distribution and failover; stable task IDs prevent duplicate zeroing on node switch or job rerun.

Reconciliation: Detect Issues, Then Controlled Repair

Periodic reconciliation checks at least three relationships:

Account available/frozen balances match ledger and batch aggregates.

Business systems and points system correspond in total and per-transaction status.

Outbox, message consumption, and downstream read models have no long-stuck records.

System generates discrepancy records and tickets with business order number, account, point amount, discrepancy type. Auto-detection ≠ auto-fix: first root-cause, then controlled补录 or reversal. Directly overwriting balance to a "correct number" breaks audit trail and may leave underlying fault running.

06 Scaling When Traffic and Data Grow

Query Volume Up: Cache and Read Models

Redis holds balance replicas, account status, hot aggregates. Historical details, leaderboards, operational reports built asynchronously into read models — data reorganized, merged, aggregated for query needs. Read models are not accounting truth and cannot decide deduct permission.

On hot cache expiry, avoid thundering herd: add small random jitter to TTLs. Block invalid params and unauthorized requests at gateway. For valid but non-existent accounts, cache empty result briefly to avoid repeated DB hits.

Core Data Volume Up: Archive First, Shard Later

Account, transaction, ledger, batch, idempotency, outbox belong in relational DB with transactions and unique constraints. Ordinary details and reports can use read replicas or async read models, but deduct decisions and authoritative balance stay on primary DB transactions.

Historical ledgers can be partitioned or moved to low-cost archive storage, provided integrity, query, and recovery processes remain verifiable. Many systems never need sharding after cold/hot tiering and index optimization.

When single DB capacity or write throughput truly hits limits, shard by stable account ID so all single-account changes stay in one shard, preserving local transactions. For huge ledger volume, partition within account shard by time or rolling tables; avoid scattering one account's data across shards by time-only routing.

If idempotency key lacks account/routing info, sharding breaks global deduplication via single-DB unique constraint. If cross-account double-entry is used, corresponding entries must share transaction boundary or accept cross-shard coordination cost.

Hot Accounts: Total Capacity ≠ No Single-Point Bottleneck

Regular user accounts rarely contend, but merchant pooled accounts, campaign accounts, or platform issuing accounts can become hotspots. Adding app instances doesn't scale writes to a single account row. Common mitigations: per-account concurrency limit, per-account sequential execution, shorten transaction duration, and where business allows, split a public account into multiple sub-accounts with aggregation.

07 Launch and Evolutionary Roadmap

Before launch, integrate stability, security, observability, cost into the same design.

Rate limit at gateway by business party, interface, account.

Set timeouts on remote calls; limited retries only for idempotent, retry-safe operations.

Size connection pools, consumer threads, batch job concurrency based on downstream throughput — don't chase message backlog and crush account DB.

Validate backup and failover via recovery drills; define RPO/RTO based on business loss.

If modules later become independently deployed services, handle service discovery, load balancing, config management, canary releases. Unify metrics, logs, traces with a single request ID, business order ID, or transaction ID. Beyond throughput, latency, error rate, monitor idempotency conflicts, account lock contention, outbox backlog, message retries, expiration job progress, reconciliation discrepancies, and alert on anomalies.

Business access requires auth, authorization, anti-replay; cross-network/org calls add signature verification per risk. High-value redemptions add risk checks. Manual adjustments need dedicated permissions, approval, immutable audit logs. Cache and read models offload primary DB but cost resources; prioritize reliability resources for core account DB, tier historical data to archive.

System evolves by real problems:

Get accounting right first : modular monolith + relational DB, local transactions close the loop on account, transaction, ledger, batch.

Move non-core work out : add cache when queries dominate; async notifications/reports when they slow main path.

Scale per bottleneck : shard when DB capacity hits ceiling; split services when teams/business lines grow; archive when history hurts online performance.

Raise recovery level : when points offset high-value benefits or cross-entity settlement, strengthen freeze/confirm, clearing, reconciliation, disaster recovery, audit.

Architecture grows this way. Every added layer must solve a problem that has already appeared or can be quantified. Don't stuff every possible component into one architecture diagram upfront. Draw clear business boundaries, core flows, data relationships first; deliver the architecture needed now; illustrate next steps in an evolution diagram with explicit triggers. Think through hard-to-change parts early (idempotency, transaction boundaries, data model); introduce cache, MQ, sharding, microservices only when real bottlenecks demand them. Think completely, land with restraint, keep evolution path ready.

08 Closing Thoughts

Revisiting that 80-point consumption: the real design challenge was never just subtraction. Business rules dictate how points are spent and refunded; data model preserves traceable facts; local transactions guard intra-account consistency; idempotency, events, reversals, and reconciliation let the system reconverge after timeouts and failures.

When evolving incrementally, you don't need to cram every conceivable component into a single architecture diagram upfront. First clarify business boundaries, core flows, and data relationships; then deliver the architecture that must land today; illustrate potential next steps in a separate evolution diagram with clear trigger conditions. The parts that are hard to change later — idempotency, transaction boundaries, data model — deserve upfront thought. Caches, message queues, sharding, and microservices should be introduced only when real bottlenecks appear. Think thoroughly, implement with restraint, and keep the evolution path prepared.