Business Compensation Mechanisms: Rollback and Retry Strategies in Distributed Systems

Hello everyone, I am Chen, an ordinary developer.

In distributed applications, a business process often composes multiple services, and a single communication may pass through DNS, network cards, switches, routers, load balancers, etc. Any unstable component can cause failures.

Under micro‑services, consistency becomes critical: if a step fails, you must either keep retrying until all steps succeed or roll back to the previous state.

We define a business compensation process as the mechanism that eliminates the inconsistent state caused by an exception.

1. About Business Compensation Mechanism

1.1 What is Business Compensation

In distributed systems, a business flow often involves a set of services, and any failure in the communication chain can lead to problems.

Micro‑service architectures make this issue more evident because business logic requires consistency guarantees.

Thus, when an operation throws an exception, we need an internal mechanism to eliminate the resulting inconsistent state.

1.2 Implementation Ways of Business Compensation Design

Business compensation can be implemented in two main ways:

Rollback (transaction compensation) : reverse operation that abandons the current step, which inevitably fails.

Retry : forward operation that attempts to complete the business flow, indicating there is still a chance of success.

Typically, transaction compensation requires a workflow engine that links various services and performs compensation to achieve eventual consistency.

Note: Compensation is an extra process; because it can be executed, timeliness is not the primary concern. The core principle is “better slow than wrong.”

2. About Rollback

Rollback means restoring a program or data to the last correct version when an error occurs. In distributed business compensation, rollback returns the system to the state before the service call.

2.1 Explicit Rollback

Rollback can be divided into two modes:

Explicit rollback : call a reverse interface to undo the previous operation or cancel an unfinished one (requires resource locking).

Implicit rollback : the downstream service automatically handles failure without extra work from the caller.

The most common is explicit rollback, which involves two steps:

Identify the failed step and its state to determine the rollback scope. If some services do not provide a rollback interface, place those that do earlier in the orchestration so they can be rolled back if later services fail.

Provide the business data needed for the rollback operation. The more data available, the more robust the program, allowing checks such as account equality or amount consistency.

2.2 Rollback Implementation

For cross‑database transactions, two‑phase commit or three‑phase commit (ACID) are common, but they are usually unsuitable for high‑availability architectures because they lock tables and degrade performance.

High‑availability systems often relax strong consistency and aim for eventual consistency, using techniques such as transaction tables, message queues, compensation mechanisms, TCC (Try‑Confirm‑Cancel), or Sagas (split transaction + compensation).

3. About Retry

Retry assumes a failure is temporary, so the operation is attempted again. This avoids the need for an extra reverse interface, reducing maintenance cost and accommodating business changes.

3.1 Retry Use Cases

Retry is suitable when downstream systems return transient errors like timeouts or rate‑limiting. It is not appropriate for permanent errors such as insufficient balance or permission denial, nor for HTTP 503/404 responses without a predictable recovery time.

3.2 Retry Strategies

Common retry strategies include:

Strategy 1 – Immediate Retry : suitable for brief glitches; should be attempted only once before switching to another strategy.

Strategy 2 – Fixed Interval : e.g., retry every 5 minutes; often used in front‑end interactions.

Strategy 3 – Incremental Interval : increase the wait time after each attempt (0 s, 5 s, 10 s, …) to prioritize newer requests.

return (retryCount - 1) * incrementInterval;

Strategy 4 – Exponential Backoff : the interval grows exponentially with each retry.

return 2 ^ retryCount;

Strategy 5 – Full Jitter : adds randomness to the exponential backoff to spread load.

return random(0 , 2 ^ retryCount);

Strategy 6 – Equal Jitter : a middle ground between exponential backoff and full jitter.

int baseNum = 2 ^ retryCount;
return baseNum + random(0 , baseNum);

Figure below shows the behavior of strategies 3‑6 (x‑axis: retry count):

3.3 Retry Precautions

Retryable interfaces must be idempotent; repeated calls should not cause cumulative changes.

Achieving idempotency involves:

Assigning a unique identifier to each request.

Checking whether the request has already been processed before executing a retry, and discarding duplicates.

Note: Retry is especially useful under high load when combined with rate‑limiting and circuit‑breaker mechanisms.

4. Precautions for Business Compensation Mechanisms

4.1 ACID vs. BASE

ACID provides strong consistency but poor scalability; BASE offers weaker consistency with better scalability and is suitable for most distributed transactions, which aim for eventual consistency rather than strict ACID guarantees.

4.2 Design Considerations for Business Compensation

Key points:

All services involved must support idempotency and upstream retry mechanisms.

The entire process should be monitored and controlled by a highly available workflow engine.

Compensation logic does not have to be a strict reverse of the forward flow; it can be parallel or simplified.

Compensation is highly business‑specific and hard to generalize.

Downstream services should provide short‑term resource reservation (e.g., reserving inventory for 15 minutes before payment).

Final Note (Please Follow)

If this article helped you, please like, view, share, and bookmark—it motivates me to keep writing.

My knowledge community is open for a fee of 129 CNY, offering Spring full‑stack, massive data sharding, DDD micro‑service series, and more. Each additional series adds 20 CNY.

To join, add my WeChat: special_coder

Follow the public account "Code Monkey Technical Column" for fan benefits and join the discussion group by replying “join”.