TCC Is Not a Silver Bullet: Engineering Go Microservice Distributed Transactions with Performance Tuning

Abstract: TCC Solves Recoverability, Not Atomic RPC

TCC (Try-Confirm-Cancel) splits cross-service actions into three local transactions : reserve, confirm, and cancel. It does not provide cross-database atomic commit; instead it uses resource reservation for business isolation, idempotent state machines, persisted decisions, and retry recovery to handle network timeouts, process crashes, and duplicate requests.

TCC suits reservable, releasable resources like balances, inventory, coupons, and seats where confirmation requires no external calls. It is unsuited for irreversible or long-running actions such as external payments, SMS, logistics, or manual approvals — those should be handled asynchronously via Outbox events or Saga/reliable messaging after order confirmation.

Before choosing TCC, answer three questions: Can Try complete provable resource reservation in a local transaction? Can Cancel safely release? Can Confirm finish using only local data? If any answer is no, do not force TCC.

1. Business Scenario: How One Order Becomes a Financial Risk

An e-commerce order must lock SKU inventory, freeze user wallet balance, and create an order. Business requires no overselling and no over-deduction; points and notifications may arrive later.

The original “deduct inventory then deduct balance, compensate on failure” is unreliable in distributed environments:

RPC timeout only means the caller is uncertain; inventory service may have already deducted. Blind retry causes double deduction.

If order service crashes after inventory success, reverse compensation never runs.

Hot SKU “check-then-deduct” races past inventory threshold, causing oversell.

Payment, points, and notifications mixed in synchronous path lengthen DB locks and tail latency.

This case limits TCC to inventory and wallet (two reservable resources). Points and notifications use Outbox events; payment gateway runs in a separate flow. This bounds lock time and keeps irreversible actions out of the compensation protocol.

1.1 Design Assumptions and Goals

Assumptions for capacity planning and load testing (actual thresholds must be calibrated): peak 10,000 TPS order placement; Try participants are inventory and wallet services; client accepts orders briefly in PROCESSING state; goal is automatic convergence after downstream recovery; if recovery window exceeded, escalate to manual audit queue.

1.2 Normal and Exception Flows

Recovery Worker  Wallet  Inventory  Order/Coordinator  Client
Recovery Worker  Wallet  Inventory  Order/Coordinator  Client
POST /v1/orders (Idempotency-Key)
Create order draft, global_tx, branches
TryReserve(tx_id, branch_id) -> RESERVED
TryFreeze(tx_id, branch_id) -> FROZEN
Persist CONFIRMING + outbox task
202 PROCESSING + tx_id
ConfirmReserve(tx_id, branch_id)
ConfirmFreeze(tx_id, branch_id)
All branches confirmed -> CONFIRMED

If any Try explicitly fails or times out, coordinator persists global decision as CANCELLING, then recovery worker sends Cancel to all branches. Timeout is not proof of remote non-execution; Cancel’s empty-rollback barrier safely handles Try not yet persisted.

2. Principles and Selection: Correct TCC Semantics

Participant state machine:

Try success
Empty-rollback Cancel
Confirm
Cancel
Repeated Cancel
Repeated Confirm
RESERVED -> CANCELLED -> CONFIRMED

CANCELLED

is Try’s terminal state: delayed Try must fail. CONFIRMED followed by Cancel is a protocol conflict — should alert, not release resources.

“Confirm/Cancel must succeed” means they are designed for safe retry until success. Confirm must not call payment gateways, send messages, or write search indexes; those are driven by Outbox records within the confirmation local transaction.

2.1 Why Not XA, Saga, or Pure Messaging

Scheme          Consistency & Isolation          Main Cost                     Applicable Boundary
XA / 2PC        Strong consistency, long DB locks  Availability & throughput drop  Few short transactions, homogeneous resources
TCC             Eventual consistency, Try isolates  Three business interfaces + state machines  Funds, inventory, coupons, seats
Saga            Eventual consistency, no reserve    Intermediate states visible, complex compensation  Long flows, many participants
Outbox / Reliable Msg  Eventual consistency, async   Consumer idempotency, reconciliation  Notifications, points, indexes, state propagation

TCC is not a superior default. If business accepts async consistency, Outbox is simpler; if no reservation semantics, forcing Try only creates states and failure points.

3. Architecture, Service Boundaries, and Data Flow

JWT + Idempotency-Key
Try, short timeout
Try, short timeout
Confirm / Cancel
Confirm / Cancel
Client -> API Gateway -> Order Service / Coordinator -> Order DB: order, global_tx, branch, outbox
Inventory Service -> Inventory DB: inventory, reservation
Wallet Service -> Wallet DB: wallet, freeze
Recovery Worker -> Outbox Publisher -> Kafka -> Points / Notification Consumers

Order Service : owns order draft, global transaction decision, branch parameters, task persistence; does not guess remote outcome from network timeout.

Inventory & Wallet Services : each maintains local ledger and reservation records; Confirm/Cancel read only local Try records, accept no mutable business parameters.

Recovery Worker : stateless horizontal scaling, claims due tasks via DB lease. Kafka only carries post-confirmation business events, not the sole store of TCC correctness.

Points & Notifications : not in TCC; consumers idempotently process by event ID.

Synchronous request only completes Try and decision, returns PROCESSING; query order for final result. This trade-off suits order APIs tolerating brief eventual consistency. If fulfillment must start immediately, synchronously wait Confirm within a fixed budget, then fall back to async query — never block indefinitely.

4. Data Model and Interface Contracts

4.1 MySQL Schema

CREATE TABLE tcc_global_tx (
  tx_id CHAR(36) NOT NULL,
  biz_key VARCHAR(96) NOT NULL COMMENT 'order no, submit idempotency key',
  decision TINYINT NOT NULL DEFAULT 1 COMMENT '1=TRYING,2=CONFIRMING,3=CANCELLING,4=CONFIRMED,5=CANCELLED',
  expire_at DATETIME(3) NOT NULL,
  next_retry_at DATETIME(3) NULL,
  retry_count INT NOT NULL DEFAULT 0,
  last_error VARCHAR(512) NULL,
  created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
  PRIMARY KEY (tx_id), UNIQUE KEY uk_biz_key (biz_key),
  KEY idx_recovery (decision, next_retry_at)
) ENGINE=InnoDB;

CREATE TABLE tcc_branch (
  tx_id CHAR(36) NOT NULL, branch_id VARCHAR(64) NOT NULL,
  participant VARCHAR(64) NOT NULL,
  state TINYINT NOT NULL DEFAULT 1 COMMENT '1=PENDING,2=TRIED,3=CONFIRMED,4=CANCELLED',
  try_payload JSON NOT NULL,
  retry_count INT NOT NULL DEFAULT 0, next_retry_at DATETIME(3) NULL,
  last_error VARCHAR(512) NULL,
  created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
  PRIMARY KEY (tx_id, branch_id), KEY idx_branch_recovery (state, next_retry_at)
) ENGINE=InnoDB;

CREATE TABLE inventory (
  sku_id BIGINT NOT NULL, available_qty INT NOT NULL,
  reserved_qty INT NOT NULL DEFAULT 0, sold_qty INT NOT NULL DEFAULT 0,
  updated_at DATETIME(3) NOT NULL, PRIMARY KEY (sku_id),
  CHECK (available_qty >= 0 AND reserved_qty >= 0 AND sold_qty >= 0)
) ENGINE=InnoDB;

CREATE TABLE inventory_reservation (
  tx_id CHAR(36) NOT NULL, branch_id VARCHAR(64) NOT NULL,
  sku_id BIGINT NOT NULL, quantity INT NOT NULL,
  state TINYINT NOT NULL COMMENT '1=RESERVED,2=CONFIRMED,3=CANCELLED',
  created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
  PRIMARY KEY (tx_id, branch_id), KEY idx_sku_state (sku_id, state)
) ENGINE=InnoDB;

In inventory, available + reserved + sold is conserved without inbound flow. Oversell prevention relies solely on atomic conditional update UPDATE ... WHERE available_qty >= ? or row lock in same transaction — never check-then-update.

4.2 External and Branch Interfaces

syntax = "proto3";
package tcc.v1;
option go_package = "example.com/tcc-demo/api/tcc/v1;tccv1";

service InventoryService {
  rpc TryReserve (ReserveRequest) returns (Ack);
  rpc ConfirmReserve (FinalizeRequest) returns (Ack);
  rpc CancelReserve (FinalizeRequest) returns (Ack);
}
message ReserveRequest { string tx_id=1; string branch_id=2; int64 sku_id=3; int64 quantity=4; }
message FinalizeRequest { string tx_id=1; string branch_id=2; }
message Ack { string code=1; string message=2; }

Order HTTP endpoint: POST /v1/orders, header must carry Idempotency-Key, server extracts user_id from JWT, never trusts body user identifier.

// request
{"sku_id":1001,"quantity":2,"amount_cent":19900}
// 202 response
{"order_id":"O202609030001","tx_id":"e0c1...","status":"PROCESSING"}

Constraints: quantity > 0, amount in minimal currency unit amount_cent > 0, SKU/order IDs validated per server-defined length/charset. Wallet amounts use BIGINT / int64 — never floating point.

5. Runnable Core: Directory, Dependencies, Config, and Inventory Implementation

Below is a standalone compilable inventory domain core. gRPC adapter only converts proto to ReserveRequest / FinalizeRequest and maps DomainError to gRPC status; ledger correctness lives entirely in this package. Uses go-sql-driver/mysql; project can split services along this structure.

tcc-demo/
├── go.mod
├── config/local.yaml
├── api/tcc/v1/inventory.proto
└── internal/inventory/service.go

// go.mod
module example.com/tcc-demo

go 1.22

require github.com/go-sql-driver/mysql v1.8.1

# config/local.yaml; real password via Secret / env var, not committed
mysql:
  dsn: "app:${MYSQL_PASSWORD}@tcp(127.0.0.1:3306)/inventory?parseTime=true&loc=UTC"
  max_open_conns: 40
  max_idle_conns: 20
  conn_max_lifetime: 30m
tcc:
  try_timeout: 300ms
  max_retry: 20

// internal/inventory/service.go
package inventory

import (
  "context"
  "database/sql"
  "errors"
  "fmt"
)

const ( reserved = 1; confirmed = 2; cancelled = 3 )

type ReserveRequest struct { TxID, BranchID string; SKUID, Quantity int64 }
type FinalizeRequest struct { TxID, BranchID string }
type Service struct { db *sql.DB }

type DomainError struct { Code, Message string }
func (e *DomainError) Error() string { return e.Code + ": " + e.Message }
func invalid(s string) error { return &DomainError{"INVALID_ARGUMENT", s} }

func NewService(db *sql.DB) *Service { return &Service{db: db} }

func withTx(ctx context.Context, db *sql.DB, fn func(*sql.Tx) error) (err error) {
  tx, err := db.BeginTx(ctx, nil); if err != nil { return err }
  defer func() { if err != nil { _ = tx.Rollback() } }()
  if err = fn(tx); err != nil { return err }
  return tx.Commit()
}

func (s *Service) TryReserve(ctx context.Context, r ReserveRequest) error {
  if r.TxID == "" || r.BranchID == "" || r.SKUID <= 0 || r.Quantity <= 0 {
    return invalid("invalid reserve request")
  }
  return withTx(ctx, s.db, func(tx *sql.Tx) error {
    var state int
    err := tx.QueryRowContext(ctx,
      `SELECT state FROM inventory_reservation WHERE tx_id=? AND branch_id=? FOR UPDATE`,
      r.TxID, r.BranchID).Scan(&state)
    switch {
    case err == nil && state == reserved:
      return nil // idempotent Try
    case err == nil && state == cancelled:
      return &DomainError{"TCC_HANGING", "branch already cancelled"}
    case err == nil:
      return &DomainError{"TCC_FINALIZED", "branch already finalized"}
    case !errors.Is(err, sql.ErrNoRows):
      return fmt.Errorf("load reservation: %w", err)
    }
    res, err := tx.ExecContext(ctx,
      `UPDATE inventory SET available_qty=available_qty-?,reserved_qty=reserved_qty+?,updated_at=UTC_TIMESTAMP(3) WHERE sku_id=? AND available_qty>=?`,
      r.Quantity, r.Quantity, r.SKUID, r.Quantity)
    if err != nil { return fmt.Errorf("reserve inventory: %w", err) }
    n, err := res.RowsAffected(); if err != nil { return err }
    if n != 1 { return &DomainError{"OUT_OF_STOCK", "insufficient stock"} }
    _, err = tx.ExecContext(ctx,
      `INSERT INTO inventory_reservation (tx_id,branch_id,sku_id,quantity,state,created_at,updated_at) VALUES (?,?,?,?,?,UTC_TIMESTAMP(3),UTC_TIMESTAMP(3))`,
      r.TxID, r.BranchID, r.SKUID, r.Quantity, reserved)
    return err
  })
}

func (s *Service) ConfirmReserve(ctx context.Context, r FinalizeRequest) error {
  return s.finalize(ctx, r, confirmed)
}

func (s *Service) CancelReserve(ctx context.Context, r FinalizeRequest) error {
  if r.TxID == "" || r.BranchID == "" { return invalid("invalid finalize request") }
  return withTx(ctx, s.db, func(tx *sql.Tx) error {
    sku, qty, state, err := load(tx, ctx, r)
    if errors.Is(err, sql.ErrNoRows) { // empty-rollback barrier, prevents delayed Try hanging
      _, err = tx.ExecContext(ctx,
        `INSERT INTO inventory_reservation (tx_id,branch_id,sku_id,quantity,state,created_at,updated_at) VALUES (?,?,0,0,?,UTC_TIMESTAMP(3),UTC_TIMESTAMP(3))`,
        r.TxID, r.BranchID, cancelled)
      return err
    }
    if err != nil { return err }; if state == cancelled { return nil }
    if state == confirmed { return &DomainError{"TCC_CONFLICT", "cannot cancel confirmed branch"} }
    res, err := tx.ExecContext(ctx,
      `UPDATE inventory SET reserved_qty=reserved_qty-?,available_qty=available_qty+?,updated_at=UTC_TIMESTAMP(3) WHERE sku_id=? AND reserved_qty>=?`,
      qty, qty, sku, qty)
    if err != nil { return err }
    n, err := res.RowsAffected()
    if err != nil { return fmt.Errorf("read release affected rows: %w", err) }
    if n != 1 { return fmt.Errorf("release inventory invariant: affected=%d", n) }
    _, err = tx.ExecContext(ctx,
      `UPDATE inventory_reservation SET state=?,updated_at=UTC_TIMESTAMP(3) WHERE tx_id=? AND branch_id=? AND state=?`,
      cancelled, r.TxID, r.BranchID, reserved)
    return err
  })
}

func (s *Service) finalize(ctx context.Context, r FinalizeRequest, target int) error {
  if r.TxID == "" || r.BranchID == "" { return invalid("invalid finalize request") }
  return withTx(ctx, s.db, func(tx *sql.Tx) error {
    sku, qty, state, err := load(tx, ctx, r)
    if errors.Is(err, sql.ErrNoRows) { return &DomainError{"TCC_PROTOCOL", "reservation not found"} }
    if err != nil { return err }; if state == target { return nil }
    if state == cancelled { return &DomainError{"TCC_CONFLICT", "reservation cancelled"} }
    res, err := tx.ExecContext(ctx,
      `UPDATE inventory SET reserved_qty=reserved_qty-?,sold_qty=sold_qty+?,updated_at=UTC_TIMESTAMP(3) WHERE sku_id=? AND reserved_qty>=?`,
      qty, qty, sku, qty)
    if err != nil { return err }
    n, err := res.RowsAffected()
    if err != nil { return fmt.Errorf("read confirm affected rows: %w", err) }
    if n != 1 { return fmt.Errorf("confirm inventory invariant: affected=%d", n) }
    _, err = tx.ExecContext(ctx,
      `UPDATE inventory_reservation SET state=?,updated_at=UTC_TIMESTAMP(3) WHERE tx_id=? AND branch_id=? AND state=?`,
      confirmed, r.TxID, r.BranchID, reserved)
    return err
  })
}

func load(tx *sql.Tx, ctx context.Context, r FinalizeRequest) (int64, int64, int, error) {
  var sku, qty int64; var state int
  err := tx.QueryRowContext(ctx,
    `SELECT sku_id,quantity,state FROM inventory_reservation WHERE tx_id=? AND branch_id=? FOR UPDATE`,
    r.TxID, r.BranchID).Scan(&sku, &qty, &state)
  return sku, qty, state, err
}

Example code must still pass go test, go vet, and real MySQL integration tests before commit; core code illustrates protocol, not a substitute for deployment, permissions, and load verification.

6. Coordinator, Retry, and Failure Recovery

Global transaction safe sequence:

Success          All succeed          Fail or timeout
Local tx: create order draft + TRYING + branches
Sequential Try
Mark branch TRIED
Local tx: decision CONFIRMING + write task
Local tx: decision CANCELLING + write task
Worker retry Confirm
Worker retry Cancel
All branches confirmed -> CONFIRMED
All branches cancelled -> CANCELLED

Key: decision persisted before remote execution . If remote Try all succeed then Confirm, then write state, a process crash leaves recovery unable to decide confirm or cancel.

func (c *Coordinator) CreateOrder(ctx context.Context, cmd CreateOrder) (txID string, err error) {
  txID = uuid.NewString()
  branches := buildBranches(txID, cmd) // inventory:sku-1001, wallet:user-42
  if err = c.repo.CreateTrying(ctx, txID, cmd.OrderID, branches, time.Now().Add(30*time.Second)); err != nil {
    if errors.Is(err, ErrDuplicateBizKey) { return c.repo.TxIDByOrder(ctx, cmd.OrderID) }
    return "", err
  }
  for _, b := range branches {
    if err = c.clients[b.Participant].Try(withTimeout(ctx, 300*time.Millisecond), b.Payload); err != nil {
      // DeadlineExceeded is unknown state; Cancel converges via empty-rollback barrier.
      return txID, c.repo.DecideAndEnqueue(ctx, txID, Cancelling, err.Error())
    }
    if err = c.repo.MarkTried(ctx, txID, b.BranchID); err != nil { return txID, err }
  }
  return txID, c.repo.DecideAndEnqueue(ctx, txID, Confirming, "")
}

CreateTrying

creates order draft, tcc_global_tx, and all tcc_branch in one local transaction; biz_key unique index guarantees API idempotency. DecideAndEnqueue uses WHERE decision=TRYING conditional update on global tx and writes task/Outbox in same transaction, preventing concurrent request decision overwrite.

Worker uses SELECT ... FOR UPDATE SKIP LOCKED or task lease to claim due records, horizontally scalable. Network errors, connection resets, Unavailable retry with exponential backoff: min(2^n × 200ms + jitter, 30s); parameter, auth, state conflicts must not blindly retry — escalate to MANUAL_REVIEW with tx, branch, error code, trace URL. Confirm failure must continue Confirm, never reverse Cancel because other branches may already be confirmed. Cancel tasks have higher priority than Confirm to avoid long resource occupation.

7. High Concurrency, Scaling, and Performance Validation

7.1 Hotspot Inventory Optimization Order

First, conditional update guarantees correctness; no cache replaces this verdict.

No RPC, Redis, or logging platform calls inside DB transaction; set MaxOpenConns, wait timeout, monitor exhaustion.

For extreme single-SKU heat, use N inventory buckets; Try picks one bucket, reservation stores bucket, Confirm/Cancel write back same bucket.

Flash-sale entry uses per-SKU token bucket and bounded queue; queue full fails fast, avoiding MySQL lock wait storms.

Redis Lua pre-deduction only as acceleration layer. Must have pre-deduction logs, failure补偿, watermark circuit breaker, and reconciliation with MySQL — MySQL remains ledger of record.

7.2 Connections, Isolation, and High Availability

Try participants ≤ 3; points, notifications moved out of TCC.

Multi-branch with limited concurrency (e.g., errgroup.SetLimit), limit set by downstream connection pools, tenant quotas, load test results.

gRPC reuses ClientConn; Try, Confirm, Cancel use independent deadlines; background tasks do not inherit user request deadline.

Bulkheads, circuit breakers, concurrency budgets for inventory and wallet. Fast reject new Try on downstream anomaly; recovery worker stays available.

Stateless service deployment, multi-replica collaboration via task leases; DB and Kafka use their own HA — do not mistake pod replica count for data HA.

7.3 Load Testing and Chaos Drills

Per Section 1 assumptions, Try phase ≈ 10,000 × 2 branches × 2 writes = 40,000 writes/s; Confirm/Cancel and fault retries extra. This is capacity estimation start, not performance guarantee. Load test must include same-SKU hotspot, 10% RPC timeout, duplicate messages, service restart, MySQL failover, MQ unavailability, coordinator rolling deploy; acceptance focuses on ledger invariants, pending convergence time, lock wait, recovery queue — not just green-path TPS.

8. Security, Observability, and Production Governance

8.1 Security Boundaries

Gateway validates JWT/OIDC; order service derives user identity from trusted claims; service-to-service uses mTLS or workload identity. tx_id, branch_id server-generated; Idempotency-Key bound to user and endpoint, preventing cross-user replay.

All SQL parameterized; order amount, quantity, SKU validated for range and business ownership.

Logs never record tokens, full bank card/account data, DB passwords; sensitive fields masked, config from Secret with rotation.

Admin retry and ledger adjustment endpoints require least privilege, dual approval, audit logs.

8.2 Metrics, Logs, and Alerts

Metric                      Dimensions                     Alert Meaning
tcc_transactions_total      decision, participant, result    Cancel surge, abnormal Confirm failure rate
tcc_pending_age_seconds     global/branch state              P99 exceeds business SLA
tcc_retry_total             participant, error_code          Retry storm or non-retryable errors
inventory_reservation_age_seconds  sku, state                RESERVED long without terminal state
db_lock_wait_seconds        table, operation                 Hotspot lock contention
outbox_lag_seconds          event_type                       Confirmation events undelivered

Structured logs at minimum include tx_id, branch_id, order_id, participant, decision, attempt, error_code. OpenTelemetry trace spans synchronous Try and async Worker; async tasks must persist and restore traceparent.

8.3 Reconciliation, Canary, and Rollback

SELECT tx_id,branch_id,sku_id,quantity,created_at
FROM inventory_reservation
WHERE state=1 AND created_at < UTC_TIMESTAMP(3) - INTERVAL 10 MINUTE;

SELECT sku_id,available_qty,reserved_qty,sold_qty
FROM inventory
WHERE available_qty < 0 OR reserved_qty < 0 OR sold_qty < 0;

Daily reconciliation combines inventory baseline, inbound/sales flow, and reservation to verify conservation. Manual handling only via audited admin commands: read global and branch state first, then retry or create approved adjustment flow; never directly modify inventory numbers.

Deploy order: “participants compatible first, then coordinator”; DDL and state enums forward-compatible. Canary watches pending age, cancel ratio, lock wait, connection pool, outbox latency. On anomaly, stop new Try first — do not stop recovery worker; pod drain: remove traffic, stop task claiming, wait in-flight deadlines, unfinished tasks taken over by lease-expiring replicas.

9. Anti-Patterns, Test Checklist, and Conclusion

High-risk anti-patterns: Confirm failure triggers Cancel; Cancel empty-rollback without barrier; Confirm/Cancel use caller-retried quantities; distributed lock replaces idempotent record; payment and notification forced into TCC; RPC timeout misjudged as Try not executed.

Pre-launch verification at minimum:

Integration tests for duplicate Try/Confirm/Cancel, Cancel-before-Try, Try timeout but server succeeded.

Conditional update oversell prevention, ledger conservation, concurrent same-SKU and multi-replica worker lease tests.

Task delivery failure, participant crash, MySQL failover, rolling deploy recovery drills.

Input validation, auth, cross-user idempotency key replay, sensitive log scanning.

Dashboards and alerts for pending, retry, lock wait, outbox, ledger invariants.

TCC’s value is not making cross-service calls “look like a database transaction”, but turning inevitable uncertainty into a persisted, retryable, observable, auditable business state machine. Only by scoping to truly reservable resources, guarding participant ledgers with local transactions, and handling long-tail faults via recovery and reconciliation, does it become engineering complexity worth bearing in Go microservices.