Distributed Systems' Hardest Challenge: State Over Interfaces — Consistency, Idempotency & Async Tasks

The Core Problem: State, Not Interfaces

The previous article discussed service evolution compatibility. This article tackles a harder problem: keeping state correct when a business operation spans multiple services. A refund flow illustrates the issue: the refund service creates an application, the payment service submits to a channel, and the order service updates after success. If the network times out after the channel accepts the request but before responding, the refund service cannot know whether to record success, failure, or retry. HTTP calls and local transactions cannot automatically resolve cross-service state, duplicate execution, or partial success.

Local Transaction Success ≠ Business Process Success

A typical refund flow involves five steps, each with its own local transaction, but no single transaction can cover the refund DB, payment DB, order DB, and external channel. Distributed transactions work only when resources are controlled and transactions are short; external channels, long-running tasks, and manual steps usually cannot join a global atomic transaction.

Three candidate approaches exist:

Local transaction in a single process — suitable when boundaries are not split, strong consistency needed, team small; low dev/recovery cost but cannot scale/deploy independently.

Cross-service synchronous calls — suitable for short chains, stable dependencies, immediate failure return, no long external actions; intuitive but timeouts and partial success make real results hard to judge.

Recoverable asynchronous flow — suitable for long processing, unstable external dependencies, need for retries and manual handling; preserves state, limits failure blast radius, enables checkpoint recovery; higher state model, observability, and ops cost.

Prefer local transactions where possible; use sync calls for low-risk short flows; only adopt async state management when the business process truly crosses multiple state and failure boundaries.

Define State Ownership First

Cross-service consistency is primarily a state ownership problem. In the refund example:

Refund service owns refund application, amount, progress, final conclusion; must not modify order records, payment transactions, channel receipts.

Payment service owns payment transactions, channel operation IDs, channel confirmation results; must not modify refund applications or order status.

Order service owns order fulfillment status and refund business impact; must not modify refund records or payment transactions.

Each service commits only within its own data boundary, exchanging facts via interfaces, events, or queries. A user-facing "refund progress" view aggregates multiple boundaries but is not an authoritative write point. Consistency does not mean all databases show the same result at the same millisecond; it requires defining which states must be immediately consistent, which can tolerate delay, how long, what facts drive convergence, and who handles boundary violations.

Separate Three State Types

Many implementations collapse everything into a single status enum (PROCESSING, SUCCESS, FAILED). This fails under timeouts because the system cannot express what it actually knows. At least three distinct state types are needed:

Business state — what stage the refund is at (accepted, processing, success, failed, pending manual).

Task state — what the executor should do next (pending, running, awaiting retry, blocked, completed).

External result — what the channel actually returned (not submitted, accepted, processing, success, explicit failure, unknown).

"Interface timeout" is a call result, not "refund failed"; "task retrying" is execution state, not a new refund creation. Merging them forces ambiguous states and special-case logic.

A proper business state machine only accepts evidence-based transitions:

Refund accepted → Channel submitting → Explicit success → Refund success
Refund accepted → Channel submitting → Explicit failure → Refund failed
Refund accepted → Channel submitting → Response timeout → Result pending confirmation
Result pending confirmation → Query confirms success → Refund success
Result pending confirmation → Confirmed rejected and no refund → Refund failed
Result pending confirmation → Long-term unknown → Pending manual confirmation

Transitions must validate pre-state and version: a successful refund cannot be reverted by a late failure message; a manually confirmed result cannot be overwritten by an old task. The three state types also need separate models: business model for rules/conclusions, persistence model for version/retry count/next execution time, exchange model for stable facts exposed to callers. Leaking retry_count, DB fields, or raw channel errors into business interfaces turns internal recovery mechanisms into external contracts.

Idempotency Identifies the Same Business Intent

Network retries, user double-clicks, task recovery, and message redelivery all cause duplicate execution. Eliminating duplicates at infrastructure level is unrealistic; the system must guarantee the same business intent expressed repeatedly produces only one effective business result.

An idempotency key can be random but must be reused for the same refund intent across retries, bound to order, amount, reason, applicant. Same key + same content returns existing refund and its current state; same key + different content rejects rather than silently overwrites.

Idempotency must cover four layers:

Business entry — one logical refund application; prevent duplicate refund records via unique constraint and existing result.

Async task — one task step or batch; prevent duplicate state advancement on restart/retry via step ID, state version, conditional update.

Message consumption — one event and its business version; prevent duplicate order modifications on redelivery via event ID or business version check.

External action — one channel refund operation; prevent duplicate fund changes on timeout by passing stable operation ID to idempotent channel; if channel only supports query, query by that ID on timeout. If channel supports neither idempotent submit nor query, the system cannot infer external non-execution from local records and must not auto-retry — preserve unknown state and escalate to manual.

Message system "exactly-once" delivery does not automatically cover DB writes, external payments, or manual ops. End-to-end business still relies on business IDs, state constraints, and reconciliation to prove no duplicate refunds.

Sync Calls, Events, and Async Tasks Solve Different Problems

These three mechanisms are often conflated but have distinct responsibilities:

Synchronous calls — for instant validation, short queries, caller must get explicit result immediately; should not wait long for channel completion or infer business failure from timeout.

Business events — notify other boundaries that a fact occurred, allowing multiple consumers to react independently; should not command all consumers to complete a single transaction in fixed order.

Async tasks — for accept-then-execute, rate-limited processing, retryable steps, long external actions; should not hide business state so users only see "background processing".

In the refund flow: synchronous validation and local acceptance, async task drives channel submission, on success publish "refund succeeded" business fact, order and notification services react per their own rules. If refund service directly sends "set order to refunded", it crosses the boundary and decides for order service.

Async is more than a task table and polling. Tasks must answer: which step is paused, what evidence the previous step left, whether next step can safely retry, what time threshold triggers blocking, and where to resume on recovery.

Local Transaction + Outbox + Checkpoints Form a Recoverable Loop

Updating refund state and publishing an event are two actions; the process may crash between them. The Outbox pattern saves business state and pending events in the same local transaction, then a separate publisher delivers and records progress. This turns an unrecoverable dual-write gap into a retryable, observable pending record — not a guarantee of exactly-once send.

The publisher may crash after sending but before recording progress, so consumers must handle duplicates idempotently. Outbox ensures consistency between business fact and pending publish record, not global transaction across all downstream states.

Long-running refund tasks also need checkpoints at key steps, persisting: refund application, current step, execution count, state version, channel operation ID, latest receipt, next execution time. On restart, the executor resumes from persisted state, not from in-memory call stack.

Platform teams can provide task scheduling, Outbox components, message delivery, retry backoff, observability; service teams still define business keys, state transitions, compensation rules, and manual escalation conditions. Platforms cannot deduce from network errors how fund business should converge; high-risk handling remains with business owners.

Eventual Consistency Is Not "It Will Be Fine Eventually"

Adopting events or async tasks often leads to a vague "eventual consistency" label. Without convergence conditions, time boundaries, and repair mechanisms, it becomes permanent inconsistency.

The refund flow must define: expected time to get channel result; which authoritative interface to query after timeout; how refund records, payment records, and channel statements are reconciled; whether differences can be auto-corrected; which discrepancies involve funds or liability judgment and require manual handling.

Compensation is not deleting a record or reverting state. Once a channel refund occurs, the system cannot roll back the database to make funds return. True compensation is a new, business-rule-constrained corrective action with its own permissions, idempotency key, state, and audit trail.

Automatic mechanisms should operate only within boundaries: retry only when error is identified as transient, operation is safely repeatable, and retry count not exceeded. Escalate to manual on: same idempotency key mapping to different business content, channel result long unknown, reconciliation mismatch, persistent state version conflicts, high-risk compensation.

Validate State Design with Failure Evidence

A passing happy path only proves the ideal route. For refund service, actively inject duplicates, delays, reordering, and interruptions to verify state still converges.

State ownership clear — each service modifies only its own state; evidence: permission checks, data lineage, architecture tests.

Duplicate execution never double-refunds — concurrent requests, task retries, message redeliveries produce one effective result; evidence: idempotency tests, unique constraints, channel reconciliation.

Task recovers from checkpoints — process exits at any step, resumes without skipping; evidence: fault injection, restart tests, task records.

Event dual-write gap recoverable — state commit leaves retryable pending publish record with delivery/consumption tracking; evidence: Outbox records, delivery logs, consumption logs.

Out-of-order and late arrivals don't overwrite new state — old events/receipts cannot roll back confirmed conclusions; evidence: chaos testing, state version records.

Unknown results are handled — query, reconciliation, manual paths have clear entry conditions; evidence: reconciliation reports, runbooks, handling records.

Unit tests verify state rules, integration tests verify component collaboration, fault injection verifies recovery, reconciliation records verify fund results. No single layer substitutes another.

Don't Turn Every Business Into an Async Workflow

When data boundaries are clear, an operation fits in one local transaction, and failure impact is limited, synchronous processing is usually more reliable. Creating tasks for simple queries, introducing full Saga without cross-service state, building event buses without authoritative facts — these are over-engineering signals.

Explicit state machines, idempotency constraints, and recoverable tasks become part of business correctness only when requests cross multiple authoritative states, external actions have unbounded latency, duplicate execution causes real loss, and the process must survive restarts.

Summary

The hardest part of distributed systems is not connecting interfaces, but knowing — at any step that may timeout, duplicate, or partially succeed — where the business stands, what facts have occurred, and whether the next step can execute safely.

Each service maintains its own authoritative state,
Every cross-boundary collaboration leaves recoverable, verifiable evidence.

Local transactions guarantee atomicity within a single boundary; idempotency constraints control duplicate execution; state machines and checkpoints enable recovery; events convey business facts; reconciliation and compensation drive eventual convergence; manual handling covers what automation cannot safely decide.