Ditch Seata: Build a Production‑Ready SAGA Orchestrator with a 500‑Line Java State Machine

Why Not Use Heavyweight Distributed‑Transaction Frameworks

Many teams immediately reach for Seata AT, TCC, transactional message queues or workflow engines, but these solutions add unnecessary infrastructure and cognitive cost when the business only needs a simple, ordered, compensable workflow. The article argues that a lightweight SAGA orchestrator built on a state machine and a relational database can satisfy most order‑fulfilment, inventory, coupon and payment scenarios.

Problem Definition

A distributed transaction should solve five concrete problems: multi‑service coordination, compensation after a failure, resilience to network glitches, idempotent handling of duplicate requests, and observability for troubleshooting.

Typical order‑fulfilment steps are:

Create order (status CREATED)

Reserve inventory (freeze stock)

Consume coupon

Create payment (status PENDING)

Failure points include order creation success followed by inventory timeout, inventory success but coupon service unavailable, payment creation succeeded but the orchestrator crashes before persisting, client retries causing duplicate orders, and compensation failures.

SAGA Fundamentals

SAGA is a long‑running transaction model that splits a global transaction into a series of local steps. Each step commits its local changes immediately; if a later step fails, previously successful steps are compensated with business‑level undo actions rather than database rollbacks.

Compensation means business‑level reversal, e.g., setting an order status to CANCELLED instead of deleting the row.

Two implementation styles exist: choreography (event‑driven, no central coordinator) and orchestration (central coordinator). The article chooses orchestration for clear, centralized state tracking, easier monitoring and simpler compensation ordering.

Overall Architecture

The system consists of an API gateway, an order application service, the SAGA orchestrator, two core tables ( saga_transaction and saga_step), downstream services (order, inventory, coupon, payment), a recovery scheduler and monitoring components.

Design Principles (Eight Must‑Haves)

Persisted state – the orchestrator never keeps progress only in memory.

Step idempotency – repeated executions must not cause side‑effects.

Compensation idempotency – retries of compensation must be safe.

Exception classification – distinguish retryable, non‑retryable and manual‑intervention errors.

Recovery capability – a crashed process can resume or compensate.

Concurrent protection – only one instance can advance a given SAGA.

Observability – the whole flow is traceable and metrics are collected.

Operability – manual retry, skip, abort and alerting are supported.

State‑Machine Design

Two state machines are defined:

Transaction‑level: NEW → RUNNING → SUCCEEDED, with branches to COMPENSATING → COMPENSATED or FAILED.

Step‑level: PENDING → EXECUTING → SUCCESS with retry states ( RETRY_WAIT) and compensation states ( COMPENSATING → COMPENSATED).

NEW -> RUNNING -> SUCCEEDED
        |
        +-> COMPENSATING -> COMPENSATED
        |
        +-> FAILED

PENDING -> EXECUTING -> SUCCESS
        |
        +-> RETRY_WAIT
        |
        +-> FAILED

PENDING -> COMPENSATING -> COMPENSATED
        |
        +-> COMPENSATE_RETRY_WAIT
        |
        +-> COMPENSATE_FAILED

Database Schema

The core tables store the transaction and step instances. All Chinese comments have been translated.

CREATE TABLE saga_transaction (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    saga_id VARCHAR(64) NOT NULL COMMENT 'Globally unique transaction ID',
    saga_type VARCHAR(64) NOT NULL COMMENT 'Transaction type, e.g., ORDER_CREATE',
    business_key VARCHAR(128) NOT NULL COMMENT 'Business idempotency key, e.g., userId:orderNo',
    status VARCHAR(32) NOT NULL COMMENT 'NEW/RUNNING/SUCCEEDED/COMPENSATING/COMPENSATED/FAILED',
    current_step INT NOT NULL DEFAULT 0 COMMENT 'Current step index',
    payload_json JSON NOT NULL COMMENT 'Snapshot of business context',
    error_code VARCHAR(64) DEFAULT NULL,
    error_message VARCHAR(512) DEFAULT NULL,
    next_retry_time DATETIME DEFAULT NULL COMMENT 'Next retry timestamp',
    version INT NOT NULL DEFAULT 0 COMMENT 'Optimistic‑lock version',
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    UNIQUE KEY uk_saga_id (saga_id),
    UNIQUE KEY uk_business_key (saga_type, business_key),
    KEY idx_status_retry (status, next_retry_time),
    KEY idx_status_updated (status, updated_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

CREATE TABLE saga_step (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    saga_id VARCHAR(64) NOT NULL,
    step_no INT NOT NULL COMMENT 'Step index, starting from 0',
    step_name VARCHAR(64) NOT NULL COMMENT 'CREATE_ORDER/RESERVE_STOCK/USE_COUPON/CREATE_PAYMENT',
    action_type VARCHAR(16) NOT NULL COMMENT 'ACTION/COMPENSATION',
    status VARCHAR(32) NOT NULL COMMENT 'PENDING/EXECUTING/SUCCESS/FAILED/RETRY_WAIT/COMPENSATING/COMPENSATED/COMPENSATE_FAILED',
    idempotency_key VARCHAR(128) NOT NULL COMMENT 'Step‑level idempotency key',
    retry_count INT NOT NULL DEFAULT 0,
    max_retry INT NOT NULL DEFAULT 3,
    last_error_code VARCHAR(64) DEFAULT NULL,
    last_error_message VARCHAR(512) DEFAULT NULL,
    next_retry_time DATETIME DEFAULT NULL,
    started_at DATETIME DEFAULT NULL,
    finished_at DATETIME DEFAULT NULL,
    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    UNIQUE KEY uk_step_once (saga_id, step_no, action_type),
    UNIQUE KEY uk_idempotency_key (idempotency_key),
    KEY idx_step_retry (status, next_retry_time),
    KEY idx_saga_step (saga_id, step_no)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Additional tables for audit logs ( saga_event_log) and external‑call records ( saga_invoke_record) are recommended for production observability.

Step Interface and Example Implementation

All steps implement a simple contract that returns a StepExecutionResult indicating success, retryability and error details.

public interface SagaStep {
    String name();
    default int maxRetry() { return 3; }
    StepExecutionResult execute(SagaContext context);
    StepExecutionResult compensate(SagaContext context);
}

Example: inventory reservation step (shows translated comments and exception handling).

@Component
public class ReserveStockStep implements SagaStep {
    private final InventoryClient inventoryClient;
    public ReserveStockStep(InventoryClient inventoryClient) { this.inventoryClient = inventoryClient; }
    @Override public String name() { return "RESERVE_STOCK"; }
    @Override public int maxRetry() { return 5; }
    @Override public StepExecutionResult execute(SagaContext ctx) {
        try {
            InventoryReserveRequest request = new InventoryReserveRequest(
                ctx.getProductId(), ctx.getQuantity(),
                ctx.stepIdempotencyKey(name(), ActionType.ACTION));
            inventoryClient.reserve(request);
            return StepExecutionResult.success();
        } catch (RemoteTimeoutException ex) {
            return StepExecutionResult.retryableFailure("INVENTORY_TIMEOUT", ex.getMessage());
        } catch (InventoryNotEnoughException ex) {
            return StepExecutionResult.nonRetryableFailure("INVENTORY_NOT_ENOUGH", ex.getMessage());
        } catch (Exception ex) {
            return StepExecutionResult.retryableFailure("INVENTORY_UNKNOWN", ex.getMessage());
        }
    }
    @Override public StepExecutionResult compensate(SagaContext ctx) {
        try {
            InventoryReleaseRequest request = new InventoryReleaseRequest(
                ctx.getProductId(), ctx.getQuantity(),
                ctx.stepIdempotencyKey(name(), ActionType.COMPENSATION));
            inventoryClient.release(request);
            return StepExecutionResult.success();
        } catch (RemoteTimeoutException ex) {
            return StepExecutionResult.retryableFailure("INVENTORY_RELEASE_TIMEOUT", ex.getMessage());
        } catch (Exception ex) {
            return StepExecutionResult.retryableFailure("INVENTORY_RELEASE_UNKNOWN", ex.getMessage());
        }
    }
}

The StepExecutionResult record provides static factories for success, retryable and non‑retryable failures.

public record StepExecutionResult(boolean success, boolean retryable, String errorCode, String errorMessage) {
    public static StepExecutionResult success() { return new StepExecutionResult(true, false, null, null); }
    public static StepExecutionResult retryableFailure(String code, String msg) { return new StepExecutionResult(false, true, code, msg); }
    public static StepExecutionResult nonRetryableFailure(String code, String msg) { return new StepExecutionResult(false, false, code, msg); }
}

Context Model

The SagaContext carries all business fields and provides helper methods to build the business key and step idempotency key.

public class SagaContext {
    private String sagaId;
    private String userId;
    private String orderNo;
    private Long productId;
    private Integer quantity;
    private String couponId;
    private BigDecimal orderAmount;
    private String traceId;
    public String businessKey() { return userId + ":" + orderNo; }
    public String stepIdempotencyKey(String stepName, ActionType actionType) {
        return sagaId + ":" + stepName + ":" + actionType.name();
    }
    // getters/setters omitted for brevity
}

Definition Registry

Each SAGA type is defined once with an ordered list of steps. The registry stores definitions in a map keyed by sagaType:version.

@Component
public class SagaDefinitionRegistry {
    private final Map<String, SagaDefinition> definitions = new HashMap<>();
    public SagaDefinitionRegistry(CreateOrderStep createOrderStep,
                                 ReserveStockStep reserveStockStep,
                                 UseCouponStep useCouponStep,
                                 CreatePaymentStep createPaymentStep) {
        SagaDefinition orderCreateV1 = new SagaDefinition(
            "ORDER_CREATE", 1,
            List.of(createOrderStep, reserveStockStep, useCouponStep, createPaymentStep));
        definitions.put(key(orderCreateV1.sagaType(), orderCreateV1.version()), orderCreateV1);
    }
    public SagaDefinition get(String sagaType, int version) {
        SagaDefinition def = definitions.get(key(sagaType, version));
        if (def == null) {
            throw new IllegalArgumentException("Saga definition not found: " + sagaType + ":" + version);
        }
        return def;
    }
    private String key(String sagaType, int version) { return sagaType + ":" + version; }
}

Core Orchestrator Logic

The orchestrator drives forward execution, handles retries, switches to compensation when a non‑retryable error occurs, and updates the transaction and step rows with optimistic‑lock (CAS) updates.

@Service
public class SagaOrchestrator {
    private final SagaTransactionRepository txRepository;
    private final SagaStepRepository stepRepository;
    private final SagaDefinitionRegistry definitionRegistry;
    public SagaOrchestrator(SagaTransactionRepository txRepository,
                            SagaStepRepository stepRepository,
                            SagaDefinitionRegistry definitionRegistry) {
        this.txRepository = txRepository;
        this.stepRepository = stepRepository;
        this.definitionRegistry = definitionRegistry;
    }
    public String start(String sagaType, int definitionVersion, SagaContext context) {
        String sagaId = UUID.randomUUID().toString();
        context.setSagaId(sagaId);
        SagaTransaction tx = SagaTransaction.newTransaction(
            sagaId, sagaType, definitionVersion, context.businessKey(), Jsons.toJson(context));
        txRepository.insert(tx);
        drive(sagaId);
        return sagaId;
    }
    public void drive(String sagaId) {
        SagaTransaction tx = txRepository.findBySagaId(sagaId)
            .orElseThrow(() -> new IllegalArgumentException("Saga not found: " + sagaId));
        if (tx.getStatus() == SagaStatus.SUCCEEDED || tx.getStatus() == SagaStatus.COMPENSATED || tx.getStatus() == SagaStatus.FAILED) {
            return; // terminal state
        }
        SagaContext ctx = Jsons.fromJson(tx.getPayloadJson(), SagaContext.class);
        SagaDefinition def = definitionRegistry.get(tx.getSagaType(), tx.getDefinitionVersion());
        if (tx.getStatus() == SagaStatus.NEW) {
            int updated = txRepository.compareAndSetStatus(
                tx.getSagaId(), SagaStatus.NEW, SagaStatus.RUNNING,
                tx.getVersion(), tx.getCurrentStep(), LocalDateTime.now(), null, null);
            if (updated == 0) return; // lost race
            tx = txRepository.findBySagaId(sagaId).orElseThrow();
        }
        if (tx.getStatus() == SagaStatus.RUNNING) {
            runForward(tx, ctx, def);
            return;
        }
        if (tx.getStatus() == SagaStatus.COMPENSATING) {
            runCompensation(tx, ctx, def);
        }
    }
    private void runForward(SagaTransaction tx, SagaContext ctx, SagaDefinition def) {
        List<SagaStep> steps = def.steps();
        int current = tx.getCurrentStep();
        while (current < steps.size()) {
            SagaStep step = steps.get(current);
            String idemKey = ctx.stepIdempotencyKey(step.name(), ActionType.ACTION);
            stepRepository.initActionStep(tx.getSagaId(), current, step.name(), idemKey, step.maxRetry());
            SagaStepInstance stepInst = stepRepository.find(tx.getSagaId(), current, ActionType.ACTION)
                .orElseThrow();
            if (stepInst.getStatus() == StepStatus.SUCCESS) {
                current++;
                tx = reloadAndAdvance(tx, current);
                continue;
            }
            boolean locked = stepRepository.updateStatus(
                tx.getSagaId(), current, ActionType.ACTION,
                stepInst.getStatus(), StepStatus.EXECUTING,
                stepInst.getRetryCount(), null, null, null) > 0;
            if (!locked) return; // another instance won the lock
            StepExecutionResult result = step.execute(ctx);
            if (result.success()) {
                stepRepository.updateStatus(
                    tx.getSagaId(), current, ActionType.ACTION,
                    StepStatus.EXECUTING, StepStatus.SUCCESS,
                    stepInst.getRetryCount(), null, null, null);
                current++;
                tx = reloadAndAdvance(tx, current);
                continue;
            }
            int nextRetry = stepInst.getRetryCount() + 1;
            if (result.retryable() && nextRetry < step.maxRetry()) {
                stepRepository.updateStatus(
                    tx.getSagaId(), current, ActionType.ACTION,
                    StepStatus.EXECUTING, StepStatus.RETRY_WAIT,
                    nextRetry, LocalDateTime.now().plusSeconds(backoffSeconds(nextRetry)),
                    result.errorCode(), result.errorMessage());
                txRepository.compareAndSetStatus(
                    tx.getSagaId(), SagaStatus.RUNNING, SagaStatus.RUNNING,
                    tx.getVersion(), current, LocalDateTime.now().plusSeconds(backoffSeconds(nextRetry)),
                    result.errorCode(), result.errorMessage());
                return; // wait for recovery job
            }
            // non‑retryable error – switch to compensation
            stepRepository.updateStatus(
                tx.getSagaId(), current, ActionType.ACTION,
                StepStatus.EXECUTING, StepStatus.FAILED,
                nextRetry, null, result.errorCode(), result.errorMessage());
            txRepository.compareAndSetStatus(
                tx.getSagaId(), SagaStatus.RUNNING, SagaStatus.COMPENSATING,
                tx.getVersion(), current, LocalDateTime.now(), result.errorCode(), result.errorMessage());
            runCompensation(txRepository.findBySagaId(tx.getSagaId()).orElseThrow(), ctx, def);
            return;
        }
        // all steps succeeded
        txRepository.compareAndSetStatus(
            tx.getSagaId(), SagaStatus.RUNNING, SagaStatus.SUCCEEDED,
            tx.getVersion(), current, LocalDateTime.now(), null, null);
    }
    private void runCompensation(SagaTransaction tx, SagaContext ctx, SagaDefinition def) {
        List<SagaStep> steps = def.steps();
        int lastSuccess = tx.getCurrentStep() - 1;
        for (int stepNo = lastSuccess; stepNo >= 0; stepNo--) {
            SagaStep step = steps.get(stepNo);
            SagaStepInstance actionInst = stepRepository.find(tx.getSagaId(), stepNo, ActionType.ACTION)
                .orElseThrow();
            if (actionInst.getStatus() != StepStatus.SUCCESS) continue; // nothing to compensate
            String idemKey = ctx.stepIdempotencyKey(step.name(), ActionType.COMPENSATION);
            stepRepository.initCompensationStep(tx.getSagaId(), stepNo, step.name(), idemKey, step.maxRetry());
            SagaStepInstance compInst = stepRepository.find(tx.getSagaId(), stepNo, ActionType.COMPENSATION)
                .orElseThrow();
            if (compInst.getStatus() == StepStatus.COMPENSATED) continue;
            boolean locked = stepRepository.updateStatus(
                tx.getSagaId(), stepNo, ActionType.COMPENSATION,
                compInst.getStatus(), StepStatus.COMPENSATING,
                compInst.getRetryCount(), null, null, null) > 0;
            if (!locked) return;
            StepExecutionResult result = step.compensate(ctx);
            if (result.success()) {
                stepRepository.updateStatus(
                    tx.getSagaId(), stepNo, ActionType.COMPENSATION,
                    StepStatus.COMPENSATING, StepStatus.COMPENSATED,
                    compInst.getRetryCount(), null, null, null);
                continue;
            }
            int nextRetry = compInst.getRetryCount() + 1;
            if (result.retryable() && nextRetry < step.maxRetry()) {
                stepRepository.updateStatus(
                    tx.getSagaId(), stepNo, ActionType.COMPENSATION,
                    StepStatus.COMPENSATING, StepStatus.RETRY_WAIT,
                    nextRetry, LocalDateTime.now().plusSeconds(backoffSeconds(nextRetry)),
                    result.errorCode(), result.errorMessage());
                txRepository.compareAndSetStatus(
                    tx.getSagaId(), SagaStatus.COMPENSATING, SagaStatus.COMPENSATING,
                    tx.getVersion(), tx.getCurrentStep(), LocalDateTime.now().plusSeconds(backoffSeconds(nextRetry)),
                    result.errorCode(), result.errorMessage());
                return;
            }
            stepRepository.updateStatus(
                tx.getSagaId(), stepNo, ActionType.COMPENSATION,
                StepStatus.COMPENSATING, StepStatus.COMPENSATE_FAILED,
                nextRetry, null, result.errorCode(), result.errorMessage());
            txRepository.compareAndSetStatus(
                tx.getSagaId(), SagaStatus.COMPENSATING, SagaStatus.FAILED,
                tx.getVersion(), tx.getCurrentStep(), null, result.errorCode(), result.errorMessage());
            return;
        }
        txRepository.compareAndSetStatus(
            tx.getSagaId(), SagaStatus.COMPENSATING, SagaStatus.COMPENSATED,
            tx.getVersion(), tx.getCurrentStep(), LocalDateTime.now(), null, null);
    }
    private SagaTransaction reloadAndAdvance(SagaTransaction tx, int nextStep) {
        txRepository.compareAndSetStatus(
            tx.getSagaId(), SagaStatus.RUNNING, SagaStatus.RUNNING,
            tx.getVersion(), nextStep, LocalDateTime.now(), null, null);
        return txRepository.findBySagaId(tx.getSagaId()).orElseThrow();
    }
    private long backoffSeconds(int retryCount) { return Math.min(60L, 1L << retryCount); }
}

The orchestrator never updates the transaction row without a version check, guaranteeing that only one instance can advance the state.

Recovery Job

A scheduled job scans for transactions in RUNNING or COMPENSATING whose next_retry_time is due, and drives them forward. ShedLock ensures only one instance runs the scan at a time.

@Component
public class SagaRecoveryJob {
    private final SagaTransactionRepository txRepository;
    private final SagaOrchestrator sagaOrchestrator;
    public SagaRecoveryJob(SagaTransactionRepository txRepository, SagaOrchestrator sagaOrchestrator) {
        this.txRepository = txRepository;
        this.sagaOrchestrator = sagaOrchestrator;
    }
    @Scheduled(fixedDelay = 3000)
    @SchedulerLock(name = "sagaRecoveryJob", lockAtLeastFor = "PT1S", lockAtMostFor = "PT2S")
    public void recover() {
        List<SagaTransaction> txList = txRepository.scanRecoverable(100, LocalDateTime.now());
        for (SagaTransaction tx : txList) {
            sagaOrchestrator.drive(tx.getSagaId());
        }
    }
}

Exception Classification

Three categories are defined:

Retryable errors (e.g., RPC timeout, network glitch, 502/503 responses) – the step status becomes RETRY_WAIT and the recovery job retries with exponential back‑off.

Non‑retryable errors (e.g., insufficient inventory, expired coupon, validation failure) – the orchestrator switches the transaction to COMPENSATING and starts reverse compensation.

Manual‑intervention errors (e.g., repeated compensation failures) – the transaction moves to FAILED, triggering alerts and allowing operators to intervene.

The orchestrator never blindly compensates on every error; it first decides whether a retry is appropriate.

Common Pitfalls and Their Remedies

Empty rollback – if compensation finds no record to undo, treat it as successful because the business state is already unchanged.

Hang‑up (悬挂) – when a forward request succeeds but the response is lost, idempotency keys and business‑level checks prevent duplicate effects.

Duplicate execution – enforce uniqueness on saga_type + business_key for the whole transaction and on idempotency_key for each step.

Compensation is not “delete” – instead of deleting rows, set logical statuses such as CANCELLED, CLOSED or AVAILABLE to keep audit trails.

High‑Concurrency Considerations

Key bottlenecks are DB writes and downstream service latency. Recommendations:

Keep DB transactions tiny – only update the transaction or step row.

Prefer optimistic locking (version column) and conditional updates over pessimistic locks.

Scan recovery tables in small batches (e.g., 50‑100 rows) and index on status + next_retry_time.

Archive old transaction rows after a retention period.

Shard tables by saga_type, tenant ID or hash of business_key if volume is very high.

API Layer

Clients start a SAGA via a REST endpoint that returns the generated sagaId. The service can be synchronous for low‑latency paths or asynchronous (client polls or subscribes) for high‑throughput scenarios.

@RestController
@RequestMapping("/api/orders")
public class OrderController {
    private final OrderSagaApplicationService applicationService;
    public OrderController(OrderSagaApplicationService applicationService) { this.applicationService = applicationService; }
    @PostMapping("/submit")
    public SagaResponse submit(@RequestBody CreateOrderSagaRequest request) {
        String sagaId = applicationService.createOrder(request);
        return new SagaResponse(sagaId, "ACCEPTED");
    }
}

The application service builds a SagaContext from the request and calls orchestrator.start("ORDER_CREATE", 1, context).

Full Failure Walk‑through

When the coupon step fails with a non‑retryable error, the transaction state transitions NEW → RUNNING → COMPENSATING. The orchestrator then compensates the previously successful steps in reverse order (release inventory, cancel order). After all compensations succeed, the transaction reaches COMPENSATED, leaving the system in a consistent state with no order, no reserved stock and no coupon consumption.

Production‑Ready Add‑Ons

Observability – expose metrics such as saga_started_total, saga_succeeded_total, saga_compensated_total, saga_failed_total, step retry counters and duration histograms.

Logging – include sagaId, sagaType, stepName, actionType, status, traceId, errorCode in every log line.

Alerting – trigger alerts when FAILED transactions appear, when compensation stays in COMPENSATING for too long, or when retry counts exceed thresholds.

Operations UI – provide a dashboard to query a transaction by sagaId, view step statuses, manually retry or mark a transaction as completed, and export audit logs.

Selection Guide

When to choose a lightweight SAGA implementation versus Seata, TCC, transactional messages or a full workflow engine:

Lightweight SAGA – fixed, short step chains (3‑8 steps), clear compensation, team can maintain core services.

Seata AT – when multiple services share the same relational database and can tolerate global locks.

TCC – when the business model naturally fits a Try/Confirm/Cancel pattern and the extra development cost is acceptable.

Transactional Message / Outbox – when eventual consistency via asynchronous events is sufficient.

Workflow Engine – for long, branching processes, visual modelling, or human approval steps.

For typical e‑commerce order flows, the lightweight SAGA is usually the most cost‑effective choice.

Checklist for Production Deployment

Use saga_type + business_key to prevent duplicate transactions.

Store step idempotency keys and enforce unique constraints.

All state updates must use version + expected status (CAS).

Index status + next_retry_time for efficient recovery scans.

Separate tables for audit logs and external‑call records.

Classify exceptions correctly and configure retry/back‑off policies.

Deploy multiple orchestrator instances with ShedLock for recovery jobs.

Expose metrics, logs and alerts; provide an ops UI for manual intervention.

Evolution Roadmap

Stage 1 – Single‑instance deployment : orchestrator, DB tables and recovery job run in one JVM.

Stage 2 – Multi‑instance scaling : share the transaction tables, use ShedLock and CAS for safe concurrency.

Stage 3 – Message‑driven advancement : after creating a transaction, publish a Kafka event; workers consume the event to drive steps, while the recovery job remains a safety net.

Stage 4 – Platformization : definition registry UI, visual workflow editor, centralized monitoring and admin console.

Conclusion

The article demonstrates that a 500‑line Java state‑machine orchestrator, backed by MySQL and Spring Boot, can satisfy production requirements for distributed transactions in typical e‑commerce scenarios. By adhering to the eight design principles, handling retries, compensation and recovery correctly, and adding observability and operational tooling, the solution is far from a toy and ready for real‑world use.