Go Distributed Transactions in Production: Outbox, SAGA, TCC & Recovery Loops
This article details production-ready Go distributed transaction patterns including Outbox for reliable event publishing, RocketMQ transaction messages, SAGA orchestration with compensation, TCC resource reservation, payment unknown-state handling, capacity planning, fault validation matrices, and operational recovery loops with concrete PostgreSQL schemas and Go code examples.
0. Conclusion First: How to Choose by Default
Don't start selection from SAGA, TCC, or XA names. First ask: where are business facts written, how to confirm after failure, and who has authority to advance state to a terminal state?
Business changes and event propagation within the same database → Default: Local transaction + Outbox. Additional capabilities: Relay, consumer idempotency, backlog recovery.
Already using RocketMQ, only need local transaction and message visibility consistency → Default: Transaction message. Additional capabilities: Immutable check credentials, version-locked SDK validation.
Cross-service process compensable, intermediate states allowed → Default: Persistent SAGA / Workflow. Additional capabilities: Step query, compensation semantics, recovery scheduling.
Must acquire resource ownership first, all participants can be modified → Default: Local TCC. Additional capabilities: Reservation ledger, anti-suspension, unique global decision.
Short transaction, controlled participants, team can handle prepared state recovery → Default: Evaluate XA. Additional capabilities: Driver, database, coordinator, and fault drills.
A typical order flow: local order + Outbox atomic commit → reliable propagation → inventory reservation → payment intent & channel reconciliation → persistent workflow advance → Outbox notify completion . Outbox and transaction messages solve reliable propagation; SAGA and TCC solve cross-service business coordination. They can combine, but a single business terminal state must have only one authoritative decision maker.
1. Consistency Is Not a Switch
1.1 Three Problems Must Be Separated
Atomic commit : Order and pending events in one local transaction either commit together or roll back together.
Business consistency : Inventory, payment, fulfillment eventually converge to allowed business states.
Completion time : Whether the API returns "accepted" or "fulfillment completed".
For example, local transaction commit only proves order and Outbox persisted, not that payment channel deducted successfully. Async acceptance should return 202 Accepted, stable order_id, and query address; all retries reuse the same request key, not generate new orders.
1.2 Write Invariants and Fact Sources First
Order : Invariant — Same tenant, same request key maps to one business intent. Authoritative fact: Order + request digest.
Inventory : Invariant — Available and reserved must not go negative; one reservation decides once. Authoritative fact: Inventory ledger, reservation records.
Payment : Invariant — One payment intent cannot retry into multiple deductions. Authoritative fact: Payment intent, channel flow, verified callback.
Message : Invariant — Committed business must have recoverable publish basis. Authoritative fact: Outbox or transaction message check credentials.
Compensation : Invariant — Every release/refund links to original op and takes effect at most once. Authoritative fact: Compensation/refund flow.
Amounts stored as integer minor currency units with currency code; quantities must be positive. All state advances verify tenant, request digest, and resource scope — never execute deduction or release solely on client-provided order ID.
1.3 Go Transaction Boundaries
*sql.Txbinds a local transaction on one connection. It does not become a distributed transaction by passing across goroutines; production code should have a single clear call chain own its lifecycle. SQL inside transaction must use tx.ExecContext / tx.QueryContext; mistakenly using db.ExecContext commits independently on another connection. Commit returning error does not prove server didn't commit: commit may have persisted while connection dropped, leaving client unconfirmed. Handle by querying fact with original request key or safe retry — never generate new business identity to bypass uncertainty.
2. Outbox: Commit "Occurred Facts" and "Facts to Propagate" Together
2.1 Data Model: Business Idempotency, Aggregate Version, Delivery Epoch Each Do Their Job
CREATE TABLE orders (
order_id text PRIMARY KEY,
tenant_id text NOT NULL,
request_key text NOT NULL,
request_hash text NOT NULL,
sku text NOT NULL,
quantity bigint NOT NULL CHECK (quantity > 0),
amount_minor bigint NOT NULL CHECK (amount_minor >= 0),
currency text NOT NULL,
status text NOT NULL,
version bigint NOT NULL DEFAULT 0,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (tenant_id, request_key)
);
CREATE TABLE outbox_events (
event_id text PRIMARY KEY,
aggregate_id text NOT NULL,
aggregate_ver bigint NOT NULL,
event_type text NOT NULL,
schema_version integer NOT NULL DEFAULT 1,
payload jsonb NOT NULL,
state text NOT NULL DEFAULT 'PENDING'
CHECK (state IN ('PENDING','INFLIGHT','PUBLISHED','DEAD')),
next_attempt_at timestamptz NOT NULL DEFAULT now(),
lease_until timestamptz,
claim_epoch bigint NOT NULL DEFAULT 0,
attempts integer NOT NULL DEFAULT 0,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz,
UNIQUE (aggregate_id, aggregate_ver, event_type)
);
CREATE INDEX outbox_pending_idx ON outbox_events(next_attempt_at, created_at, event_id)
WHERE state = 'PENDING';
CREATE INDEX outbox_expired_idx ON outbox_events(lease_until, event_id)
WHERE state = 'INFLIGHT';request_keyprevents client retries creating multiple orders; request_hash rejects "same key, different amount or SKU" misuse. aggregate_ver is the aggregate's business change count — must not be hardcoded to 1 ; claim_epoch is the Relay's lease epoch for the same event, not for business ordering.
2.2 Atomic Creation and Aggregate Version Generation
Creating an order sets initial version to 1; subsequent order state changes in the same transaction obtain next version via conditional update, then write corresponding Outbox event. The example deliberately separates "duplicate request" from "new request": unique key conflict aborts PostgreSQL transaction, must rollback then query original order.
func CreateOrder(ctx context.Context, db *sql.DB, in OrderInput) (Order, error) {
payload, err := json.Marshal(OrderCreated{OrderID: in.OrderID, SKU: in.SKU, Qty: in.Quantity})
if err != nil { return Order{}, fmt.Errorf("marshal event: %w", err) }
tx, err := db.BeginTx(ctx, nil)
if err != nil { return Order{}, err }
defer tx.Rollback()
_, err = tx.ExecContext(ctx, `
INSERT INTO orders(order_id, tenant_id, request_key, request_hash, sku, quantity,
amount_minor, currency, status, version)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,'PENDING',1)`,
in.OrderID, in.TenantID, in.RequestKey, in.RequestHash, in.SKU,
in.Quantity, in.AmountMinor, in.Currency)
if err != nil { return Order{}, classifyCreateOrderError(err) }
_, err = tx.ExecContext(ctx, `
INSERT INTO outbox_events(event_id, aggregate_id, aggregate_ver, event_type, payload)
VALUES ($1,$2,1,'OrderCreated',$3::jsonb)`, in.EventID, in.OrderID, string(payload))
if err != nil { return Order{}, fmt.Errorf("insert outbox: %w", err) }
if err = tx.Commit(); err != nil { return Order{}, fmt.Errorf("commit uncertain: %w", err) }
return Order{ID: in.OrderID, Status: "PENDING"}, nil
}For existing order state changes, use pattern:
UPDATE orders
SET status = $3, version = version + 1
WHERE order_id = $1 AND version = $2
RETURNING version;Read returned version, then insert corresponding event in the same transaction . RowsAffected=0 means stale write or illegal state; caller should read current state and retry with original business identity or reject — never blindly overwrite.
2.3 Duplicate Requests and Unknown Commit Result
classifyCreateOrderErroronly marks unique conflict as "need to query original", cannot treat all unique conflicts as success. After rollback, read order by (tenant_id, request_key): if request digest matches, return same order; if mismatch, return conflict. If Commit errors, same query confirms fact.
Key boundary: Business idempotency key solves duplicate intent; event ID solves message redelivery; they cannot substitute each other.
2.4 Relay: Short Transaction Claim, Send Outside Transaction, Conditional Write-Back
WITH picked AS (
SELECT event_id FROM outbox_events
WHERE state = 'PENDING' AND next_attempt_at <= now()
ORDER BY next_attempt_at, created_at, event_id
LIMIT $1 FOR UPDATE SKIP LOCKED
)
UPDATE outbox_events e
SET state = 'INFLIGHT', lease_until = now() + ($2 * interval '1 second'),
claim_epoch = e.claim_epoch + 1, attempts = e.attempts + 1
FROM picked
WHERE e.event_id = picked.event_id
RETURNING e.event_id, e.event_type, e.payload, e.claim_epoch;In short transaction, read all rows from RETURNING and close result set, then send to Broker. After Broker ack, write back by event ID and claim epoch:
UPDATE outbox_events
SET state = 'PUBLISHED', published_at = now(), lease_until = NULL
WHERE event_id = $1 AND state = 'INFLIGHT' AND claim_epoch = $2;Must check affected rows. Zero rows means lease recycled or state advanced by new Relay; old Relay must stop writing. Epoch cannot prevent late send after lease expiry, so system remains at-least-once; consumers must be idempotent.
Publish failure also conditionally writes back to PENDING using event_id + claim_epoch, with capped exponential backoff and jitter; exceed limit → DEAD and alert. Expired lease recovery also uses FOR UPDATE SKIP LOCKED in batches. Batch size, concurrency, and lease must match: if second half hasn't sent before lease expires, it actively creates duplicate delivery.
A Go Relay core is separating DB transaction boundary from network call boundary — not putting MQ client inside tx:
type ClaimedEvent struct {
ID, Type string
Payload []byte
Epoch int64
}
func relayOnce(ctx context.Context, db *sql.DB, broker Broker) error {
events, err := claimEvents(ctx, db, 100, 30*time.Second) // internal short tx
if err != nil { return err }
for _, e := range events {
// send with stable event_id; consumer and broker partition key can use aggregate_id
err := broker.Publish(ctx, Message{Key: e.ID, Type: e.Type, Body: e.Payload})
if err == nil {
if err = markPublished(ctx, db, e.ID, e.Epoch); err != nil { return err }
continue
}
if err = reschedule(ctx, db, e.ID, e.Epoch, classifyPublishError(err)); err != nil {
return err
}
}
return nil
}markPublished, reschedule must use state = 'INFLIGHT' AND claim_epoch = $epoch as condition and check affected rows. Broker timeout cannot be safely classified as "not published": usually publish result unknown, must treat as at-least-once. Only auth failure, malformed message etc. permanent errors suit fast manual handling; others do bounded retry first.
2.5 Event Contract, Evolution, and Isolation
Events must carry at least event_id, event_type, schema_version, aggregate_id, aggregate_ver, occurrence time, and trace correlation ID. Don't let consumers depend on DB internal fields or serialize entire order row as permanent contract. Field additions usually backward compatible; deletions, semantic changes, or currency unit changes require version bump, dual read/emit, or consumer migration.
Producer publishes facts (e.g., OrderCreated, InventoryReserved), not commands like "please deduct inventory". Commands need explicit receiver and response/retry semantics; events allow multiple subscribers each maintaining projections. Mixing them makes one topic carry both orchestration and audit, eventually unable to define replay meaning.
For events with privacy or financial sensitivity, payload keeps only business-necessary fields; sensitive data passed via controlled reference or masked. Message ACL, transport encryption, signature/auth, audit retention are also part of consistency system. Untraceable "idempotent messages" can still cause correct execution of wrong operation.
2.6 Consumer: Inbox and Business Effect in Same Transaction
CREATE TABLE inbox_events (
consumer_name text NOT NULL,
event_id text NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_name, event_id)
);Consumer local transaction order: insert Inbox (with ON CONFLICT DO NOTHING) → if not inserted, exit → execute business conditional update and check affected rows → commit then ACK. If commit succeeds but ACK lost, redelivery caught by Inbox; if business fails, Inbox insert rolls back with transaction.
Inbox only removes duplicate processing of same message. For inventory reservation, refund etc., still need business unique keys to avoid two different event_id from upstream causing two business effects. External HTTP payment cannot be wrapped in DB transaction; still rely on payment intent, channel idempotency key, and query reconciliation.
Business updates should be explicit state transitions, e.g.,
UPDATE reservations SET status='TRIED' WHERE biz_key=$1 AND status='INIT', not "on message received overwrite to target value". Former distinguishes duplicate, out-of-order, illegal state, and genuine retry-needed failure; latter masks protocol errors.
2.7 Ordering, Retention, and Recovery
created_atorder is not arrival order. Concurrent Relays, retry redelivery, and partitioned messages all reorder. Partition by aggregate key, and consumer verifies aggregate_ver: on gap, wait, backfill, or rebuild projection — never apply high version and forget low version's business action.
Inbox retention must cover message retention, historical replay, and manual redelivery window; purging records may let replayed old messages take effect again. Polling at 1 second adds ~0.5s average extra latency when no backlog, plus query, send, consume time. LISTEN/NOTIFY can wake, but persistent Outbox and periodic rescan remain recovery basis.
3. RocketMQ Transaction Message: Bind Check-Back and Immutable Facts
Transaction message goal: "local transaction result and message visibility eventually consistent", not "downstream completed". Producer sends half-message, executes local transaction, then commit/rollback decides message visibility; on network anomaly Broker checks back.
3.1 Establish Stable Identity for Check-Back
Half-message, business event, and local credential use stable tx_id / event_id, and write business change plus COMMITTED credential in local transaction:
CREATE TABLE message_tx_receipts (
tx_id text PRIMARY KEY,
event_id text NOT NULL UNIQUE,
request_hash text NOT NULL,
outcome text NOT NULL CHECK (outcome IN ('COMMITTED','ABORTED')),
created_at timestamptz NOT NULL DEFAULT now()
);Check-back reads only primary DB or other authoritative source: matching COMMITTED credential → Commit; mutually exclusive ABORTED terminal state → Rollback; DB unavailable, incomplete evidence, or original transaction possibly executing → return Unknown and monitor. No record found must not directly Rollback. ABORTED cannot be casually补写 by a timeout task: it must share same conditional state transition with original execution, making original transaction impossible to late-commit. Otherwise race: one side rules rollback, other commits business fact. Credential retention must cover Broker check-back and audit window.
3.2 Transaction Message Implementation Checkpoints
Different RocketMQ server and Go SDK transaction message interfaces, check-back concurrency, and max retries change; at launch lock specific versions and confirm via integration tests:
How business key for half-message enters local transaction callback, and how retrieved at check-back; forbid generating another ID temporarily.
Process exit before local commit, response drop after commit, check-back concurrent with local transaction — does check-back only read authoritative source?
What happens when Unknown reaches Broker configured limit: alert, manual verify, or Broker discard; cannot assume infinite retry.
Producer transaction message final visibility ≠ consumer completion; consumer still needs Inbox, business key, and dead-letter recovery.
Transaction messages and Outbox are not mutually exclusive "sect choices". They solve similar core windows but differ in ops and dependencies: Outbox keeps facts entirely in DB and app publishes; transaction message hands half-message and check-back semantics to Broker. For same domain event, pick one primary recovery channel; if both coexist, must draw precise responsibility boundaries to avoid dual publish creating two retry sources.
Same DB change to same destination event: usually choose one of Outbox or transaction message as primary publish recovery mechanism; if combined, must clarify which handles publish, which handles which state, and reuse same event identity.
4. SAGA: Compensation Is Not Rollback, But New Auditable Action
SAGA coordinator must persist flow ID, step ID, input digest, business idempotency key, forward and compensation status, retry time, and global decision. In-process goroutine, defer, or errgroup do not constitute recovery capability.
Most dangerous step is not explicit failure but "service committed, success response lost". Coordinator for unknown result can only query authoritative fact, or call compensation interface with idempotency, empty compensation, and anti-suspension; cannot compensate only steps that received success response.
Compensating order A's three inventory reservations must operate on A's reservation flow, not overwrite total inventory to old snapshot — else erases order B's legitimate changes. Payment refund is independent side effect: has refund order, original payment link, refund key, and UNKNOWN state — cannot describe as time reversal.
DTM-like frameworks provide persistent coordination and sub-transaction barriers; correct integration covers idempotency, empty compensation, suspension within protocol scope; external HTTP, side effects, auth, business semantics, compensation feasibility remain business responsibility. Independent steps can run parallel but must persist dependency graph; one branch fails while another in-flight → first converge in-flight and unknown results, then determine compensation scope.
4.1 Model Step Result as "Unknown" Not Boolean
Succeeded : Coordinator can advance successors, keep compensation basis. Should not assume never need to query again.
Failed : Coordinator can decide compensation per dependency graph. Should not send compensation for not-yet-started steps.
Unknown : Coordinator can query participant fact, or call safe compensation per protocol. Should not deem failed just because RPC timeout.
Compensated : Coordinator can record compensation flow and continue convergence. Should not treat compensation as original op never happened.
Manual Review : Coordinator can freeze auto decision, preserve evidence. Should not cron job unconditionally force close.
Example: "reserve inventory → create payment intent → payment confirm → ship". Inventory reserved but response lost → coordinator first queries by reservation business key; query fails then controlled recovery. Payment succeeded but ship fails → usually not "cancel payment" but create refund flow, which itself may be unknown. Irreversible ops like shipping, SMS, coupon grant need business remediation (intercept, revoke coupon, CS handling) — not claim universal compensation.
4.2 Minimum Fields for SAGA Recovery Record
Suggest at least: gid, step name, branch business key, input digest, attempt count, next retry time, forward status, compensation status, last evidence, global decision version. Each scheduling worker uses lease and epoch conditional update when claiming step to prevent two recoverers concurrently advancing same step. Global flow state is orchestration view, must not replace inventory ledger or payment channel facts.
5. TCC: Resource Ledger and State Machine Must Commit in Same Transaction
Try is not "try and see" but commits resource reservation fact; Confirm consumes reservation; Cancel releases reservation. Suits reservable resources, not forcing wrap of payment channels that don't support freeze+confirm.
CREATE TABLE inventory (
sku text PRIMARY KEY,
available bigint NOT NULL CHECK (available >= 0),
reserved bigint NOT NULL DEFAULT 0 CHECK (reserved >= 0)
);
CREATE TABLE inventory_reservations (
gid text NOT NULL,
branch_id text NOT NULL,
biz_key text NOT NULL UNIQUE,
request_hash text NOT NULL,
sku text NOT NULL,
qty bigint NOT NULL CHECK (qty > 0),
status text NOT NULL CHECK (status IN ('INIT','TRIED','CONFIRMED','CANCELLED')),
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (gid, branch_id)
);5.1 Executable Conflict Classification Flow
Try, Confirm, Cancel must carry coordinator-persisted immutable fields: gid, branch_id, biz_key, request digest, SKU, quantity, tenant scope. All resource changes and status changes in one local transaction.
Attempt insert
(gid, branch_id)INITrecord.
If PK exists, lock that record in same transaction, compare immutable fields item by item: all equal → allow idempotent handling per existing state; any mismatch → reject.
If biz_key unique conflict aborts PostgreSQL transaction, immediately rollback; in new transaction read existing record by biz_key, compare fields, return explicit "cross-branch business conflict" — not treat as success.
Record absent but Cancel received → write CANCELLED placeholder with full immutable fields; late Try reads that state and rejects, forming anti-suspension. ON CONFLICT (gid, branch_id) DO NOTHING only handles PK conflict, does not swallow biz_key UNIQUE conflict — a PostgreSQL detail many examples miss.
5.2 State Machine and Inventory Update
Not exist : Try → create record and reserve; Confirm → protocol error; Cancel → write full cancel placeholder.
TRIED : Try → same-param idempotent success; Confirm → consume reservation, become CONFIRMED; Cancel → release reservation, become CANCELLED.
CONFIRMED : Try → forbid re-reserve; Confirm → idempotent success; Cancel → decision conflict, reject and alert.
CANCELLED : Try → reject late Try; Confirm → decision conflict; Cancel → idempotent success.
Try executes in local transaction locking reservation record:
UPDATE inventory
SET available = available - $2, reserved = reserved + $2
WHERE sku = $1 AND available >= $2;Must confirm one row affected, then change INIT to TRIED and update updated_at, then commit. Confirm only transitions TRIED → CONFIRMED, same transaction decrements reserved; Cancel only TRIED → CANCELLED, same transaction available += qty, reserved -= qty. Forbidden to split status table and Freeze into two transactions.
Multi-SKU ops lock in stable order; on deadlock or serialization failure retry with original business identity. Coordinator persists unique global decision; after Confirm decision, participants can only retry Confirm or query, not auto-Cancel on timeout. Reservation timeout task must verify authoritative coordinator state, not release inventory solely by age.
5.3 Common TCC Pitfalls
Try in Redis, Confirm in DB : Lua only guarantees Redis internal atomicity, cannot make Redis and PostgreSQL commit together; failover, TTL, cross-storage crash window, reconciliation remain.
Read inventory then deduct : concurrent reads oversell. Use conditional update or equivalent row lock, check affected rows.
Empty rollback writes empty business key : different branches conflict. Cancel placeholder must save full comparable request identity.
Release by reservation age directly : late Confirm may face re-sold inventory. Release must be constrained by global decision or controlled timeout protocol.
Only final status field : without resource flow and audit fields, cannot explain what a Confirm/Cancel changed, hard to reconcile.
Seata TCC Fence shares similar goal with above control records. If using framework, verify version, storage, transaction wrapping; don't build another fence with different meaning overlapping outside framework.
6. Payment Unknown: External Side Effects Lack DB-Style Atomic Commit
Payment timeout only means caller unknown — request may not have arrived, channel failed, or channel succeeded but response lost. Persist payment_intent_id, order, channel, amount, currency, merchant request key, channel flow; merge retries, queries, callbacks into same payment intent.
CREATED → PROCESSING → SUCCEEDED / FAILED / UNKNOWN → REVIEW → REFUNDING → REFUNDEDCallback must verify signature and cross-check merchant, amount, currency; use conditional state transition to prevent old failure callback overwriting success. If channel supports idempotency, retry reuses original key respecting its scope and TTL; if not idempotent and cannot reliably query, no local solution can create "exactly-once deduction" — limit auto retry and escalate to manual review. Refund may also be UNKNOWN; cannot create another refund order on timeout.
7. Capacity, Fault Acceptance, and Operational Loop
7.1 Metrics Must Align with Completion Boundaries
Separate reporting: acceptance TPS (local fact committed), business completion TPS (success or explicit failure terminal), successful fulfillment TPS (inventory and payment both confirmed), recovery throughput. Compensated failures count as terminal, not successful fulfillment.
If ingress λ, publish pause t seconds, backlog ≈ λ × t; recovery capacity μ with unchanged new traffic, only μ > λ drains, ideal drain time ≈ λt / (μ - λ). This is capacity estimate; real drills must cover retries, hot SKUs, MQ and DB bottlenecks.
Stress test report must fix DB, Broker, SDK versions, persistence/replication config, connection pool, CPU/memory limits, network topology, test duration, warmup, timeout and retry budgets. Uniform SKU and hot SKU tested separately.
7.2 Minimum Fault Matrix
Order submit then immediate exit : Order and Outbox both exist, Relay recoverable.
Broker ack then exit before write-back : Allow duplicate messages, but business effect only once.
Lease expired, old Relay late : Old epoch cannot overwrite DB state.
Consumer commit then ACK lost : Inbox and business key prevent duplicate effect.
Transaction message check-back DB unavailable / transaction in-flight : No misjudge Rollback, enter Unknown and alert.
SAGA success response lost : Query fact or safe compensation then converge.
Cancel arrives first, Try late : Cancel placeholder blocks reservation.
TCC same key diff params, Confirm/Cancel conflict : Explicit reject, keep terminal state, alert.
Payment success but call timeout : Query or callback merge into original payment intent.
Coordinator, Worker, DB primary/standby switch : Recover from persisted state and complete reconciliation.
These tests need real DB transactions and controllable fault points. Only mocking SQL calls cannot prove consistency.
7.3 PostgreSQL and DB Ops Notes
SKIP LOCKEDsuits work queue claiming, not general consistency queries. Relay claiming typically uses READ COMMITTED; for business transactions relying on read-write dependencies, specify isolation level, treat deadlock and serialization failure as transient errors retryable with same business identity. Retry must have budget, backoff, deadline to avoid amplifying DB pressure during faults.
Partial indexes only accelerate active state scans, don't auto-solve table bloat. Outbox, Inbox, reservation, audit tables need archival and cleanup rates; observe autovacuum, index bloat, long transactions, replication lag. Before cleanup confirm message replay, late requests, Broker check-back, financial audit windows; data recovery must also trigger external payment and inventory reconciliation — DB backup restore is not complete business recovery.
7.4 Metrics, Alerts, and Manual Handling
Metrics use low-cardinality labels: Outbox pending count and oldest age, dead-letter increment, lease recovery count; Inbox dedup count and consume failures; SAGA non-terminal age; TCC pending reservations and terminal conflicts; payment Unknown age, pending refunds, reconciliation amount diff. Business IDs only in logs and traces, not Prometheus labels.
Runbook turns metrics into actions: oldest Outbox age reaches 50% business SLA → warning, exceeds SLA → alert; DEAD or terminal conflict increment → immediate alert; payment UNKNOWN exceeds channel query window → create manual review ticket. Manual console only issues controlled recovery commands, records evidence, operator, before/after state and result — never directly edit tables bypassing ledger.
Reconciliation scans timeout non-terminal orders, reads inventory reservations, payment and channel evidence, flow steps, publish credentials, then generates recovery commands per legal state transitions. For external side effects, DB conditional update insufficient for dedup; still must use channel idempotency key and query protocol.
7.5 Kubernetes and Disaster Recovery
Multiple replicas don't automatically bring transaction recovery. Coordinator needs reliable persistence and recoverable scheduling; participants need stable callback addresses, auth, timeouts, pool config. Graceful shutdown stops claiming new tasks, allows in-flight requests to complete within grace period, remaining tasks recovered via lease/persisted state. Pod restart, network partition, DB primary/standby switch, Broker unavailability, backup restore must be in drill plan.
Config center, service discovery, transaction log, business DB serve different roles; any component "has three replicas" cannot replace another's recovery strategy. Especially point-in-time recovery may let external deduction happen while local payment intent rolls back — after recovery must prioritize diff reconciliation.
8. Launch Checklist
Same request retried 10 times consecutively — always points to same order, payment intent, and business effect?
Any commit response lost — which authoritative table or channel interface confirms fact?
Old messages, old requests, expired leases, out-of-order actions — can they re-trigger effects?
After Confirm decision, any other worker or cleanup task might release resources unauthorized?
After dedup record cleanup, historical replay, backup restore — how does system reject expired identities or complete reconciliation?
When auto recovery stalls, who receives alert, what evidence does manual use to execute which controlled command?
Production standard for distributed transactions is not zero failures, but every success has evidence, every unknown has a destination, every retry reuses same identity, and all terminal states withstand duplication, reordering, and process restart.
References and Validation Scope
Protocol boundaries subject to actual deployed versions; complete integration and fault tests before release:
Go: Executing transactions
Go: database/sql
PostgreSQL: SELECT / SKIP LOCKED
RocketMQ: Transaction Message
Apache Kafka: Design
DTM: Sub-transaction Barrier
Apache Seata: TCC Fence
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
