Why Stale Workers Still Cause Failures: Lease, Fencing & Outbox Patterns for Go + PostgreSQL

01 The Lock Didn't Fail, But the Old Worker Is Still Running

Worker A holds a 30-second lease, then gets paused (e.g., SIGSTOP, GC pause, VM freeze). The lease expires, Worker B takes over with a new epoch. When A resumes, its in-memory state, goroutines, and in-flight HTTP requests remain active. Even with correct Redis SET NX PX or PostgreSQL conditional updates, A can still:

Re-send create requests to third parties

Overwrite B's saved checkpoints

Mislabel B's continued task as failed or succeeded

A lease only declares "who the coordination system currently recognizes"; it cannot reclaim CPU from a paused process or cancel remote requests already received. The correct goal is not "only one worker computes at a time" but four invariants:

Only the current run's current epoch may write workflow state.

State progression within the same epoch must have ordering constraints so late arrivals cannot overwrite newer results.

One business intent always uses the same external operation identity; takeover and retry must not generate new identities.

When remote results are unknown, the system preserves evidence and enters query/reconciliation/manual handling — never treats timeout as failure.

A timeline illustrates the scenario: at t0 A gets epoch=100; t1 A pauses; t2 lease expires, B gets epoch=101; t3 A resumes and attempts commit but finds current epoch is B/101. From t3 onward A can burn CPU but cannot cross the database commit boundary; its in-flight remote calls are handled by idempotency and reconciliation protocols.

02 Distinguish Four Identities — Don't Overload One ID

The article defines five key fields:

task_id : User-visible logical task. Stable across retries. Must not be used as isolation boundary for stale requests.

run_id (UUID/ULID): Single non-reusable execution instance. New on every retry. Must not be guessed or reused by caller. Critical: retries must not reuse task_id because old HTTP requests, messages, or tool results may arrive hours later and pollute the new execution. Carry run_id in external requests, ledgers, logs, and callbacks.

owner : One worker execution session. Changes on each acquire/takeover. Must not be used as fencing total-order token.

epoch : Ownership term within a run. Increments on successful takeover. Must not be used for cross-run or cross-resource ordering. Epoch is a fencing token but only ordered within the same run_id. Job-A epoch 10 and Job-B epoch 20 are incomparable; if two tasks write the same account or shared resource, establish version, unique constraint, or lock domain on that real resource.

revision : State version within a run. Increments on each state write. Must not be used as external side-effect identity.

operation_id : One business side-effect intent. Constant for the same intent. Must not be used as executor, retry count, or timestamp identity.

03 Redis Locks Coordinate, But Cannot Replace Commit Constraints

Redis safe release/renewal must compare a random owner token:

SET task:lock:42 <random-owner-token> NX PX 30000

-- release / renew: compare value first; renew ARGV[2] is TTL ms
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('PEXPIRE', KEYS[1], ARGV[2])
end
return 0

Random tokens prevent deleting/renewing someone else's lock, but they lack sortable semantics and cannot serve as fencing tokens. Renewal improves availability but cannot revoke in-flight network requests. Re-reading the lock before a business step is still unsafe: a race exists between check and commit.

Redis can be a contention pre-filter, not the sole safety boundary. If Redis is unavailable, bypassing the pre-filter is a capacity policy; regardless, PostgreSQL protected-write rules must always hold. Long pauses are not only Go STW GC — container freezes, VM pauses, CPU starvation, network stalls, and scheduler latency can all cause missed heartbeats.

04 Data Model: Separate Logical Task, Run Instance, Lease, and Side-Effect Ledger

Schema for PostgreSQL 16 (all safety decisions hit the single writable primary; default isolation READ COMMITTED; treat SQLSTATE 40001 as "retry the whole short transaction"):

CREATE TABLE agent_tasks (
  task_id      text PRIMARY KEY,
  active_run_id uuid,
  created_at   timestamptz NOT NULL DEFAULT clock_timestamp(),
  updated_at   timestamptz NOT NULL DEFAULT clock_timestamp()
);

CREATE TABLE agent_runs (
  run_id       uuid PRIMARY KEY,
  task_id      text NOT NULL REFERENCES agent_tasks(task_id),
  status       text NOT NULL CHECK (status IN ('pending','running','succeeded','failed','cancelled')),
  revision     bigint NOT NULL DEFAULT 0 CHECK (revision >= 0),
  checkpoint   jsonb NOT NULL DEFAULT '{}'::jsonb,
  created_at   timestamptz NOT NULL DEFAULT clock_timestamp(),
  updated_at   timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX agent_runs_task_created_idx ON agent_runs(task_id, created_at DESC);

CREATE TABLE worker_leases (
  run_id       uuid PRIMARY KEY REFERENCES agent_runs(run_id),
  owner        text,
  epoch        bigint NOT NULL DEFAULT 0 CHECK (epoch >= 0),
  expires_at   timestamptz NOT NULL DEFAULT 'epoch',
  updated_at   timestamptz NOT NULL DEFAULT clock_timestamp()
);

CREATE TABLE tool_operations (
  operation_id   text PRIMARY KEY,
  run_id         uuid NOT NULL REFERENCES agent_runs(run_id),
  step_id        text NOT NULL,
  effect_version integer NOT NULL CHECK (effect_version > 0),
  tool_name      text NOT NULL,
  request_digest text NOT NULL,
  request_body   jsonb NOT NULL,
  status         text NOT NULL CHECK (status IN ('prepared','dispatching','unknown','succeeded','failed_final','cancelled')),
  dispatch_owner text,
  dispatch_until timestamptz,
  remote_id      text,
  result         jsonb,
  revision       bigint NOT NULL DEFAULT 0,
  updated_at     timestamptz NOT NULL DEFAULT clock_timestamp(),
  UNIQUE (run_id, step_id, effect_version)
);
CREATE INDEX tool_operations_reconcile_idx ON tool_operations(status, dispatch_until, updated_at);

Creating task, run, and lease must be in the same short transaction; switching active_run_id happens there too. Old runs are never reused or deleted beyond auditability; archive after retention per compliance. If external systems need their own fencing, the high-water key must be at least (resource_id, run_id), not bare epoch.

05 Lease Protocol: Single Row Serializes Terms

5.1 Acquire or Takeover

Caller validates TTL > 0 and sets platform ceiling; pass integer milliseconds, not Go Duration.String() as PostgreSQL interval.

-- $1=run_id, $2=owner, $3=ttl_ms
UPDATE worker_leases
SET owner = $2,
    epoch = epoch + 1,
    expires_at = clock_timestamp() + ($3::bigint * interval '1 millisecond'),
    updated_at = clock_timestamp()
WHERE run_id = $1
  AND (owner IS NULL OR expires_at <= clock_timestamp())
RETURNING run_id, owner, epoch, expires_at;

Zero rows may mean run doesn't exist, not yet expired, or held. For APIs, query additionally to distinguish, but don't treat "failed to acquire lease" as "task failed". Concurrent updates on the same row make PostgreSQL wait and re-evaluate the condition after the lock releases — this is the behavior the protocol relies on.

5.2 Renewal and Release

-- renew: $1=run_id, $2=owner, $3=epoch, $4=ttl_ms
UPDATE worker_leases
SET expires_at = clock_timestamp() + ($4::bigint * interval '1 millisecond'),
    updated_at = clock_timestamp()
WHERE run_id = $1 AND owner = $2 AND epoch = $3 AND expires_at > clock_timestamp();

-- release: do NOT zero epoch
UPDATE worker_leases
SET owner = NULL, expires_at = clock_timestamp(), updated_at = clock_timestamp()
WHERE run_id = $1 AND owner = $2 AND epoch = $3;

Renewal returning zero rows means lease lost — cannot resurrect. Network timeout is not proof of renewal failure: DB may have updated while response was lost. Conservative default: cancel current execution, stop dispatching new side effects, let lease expire naturally. If product wants to resume the same session, must first re-confirm from primary that same owner + epoch still valid.

06 Protected State Commit: Lock Lease, Then Validate, Then CAS Run

High-water fencing only rejects old tokens after observing a larger one; it cannot auto-detect a lease that just expired. Therefore workflow state in the database uses "current lease validation + same-transaction update" as a stronger boundary.

var (
  ErrLeaseLost = errors.New("lease lost")
  ErrConflict  = errors.New("run revision conflict")
)

type Lease struct {
  RunID uuid.UUID
  Owner string
  Epoch int64
}

func SaveCheckpoint(ctx context.Context, db *pgxpool.Pool,
  l Lease, expectedRevision int64, checkpoint []byte,
) (int64, error) {
  tx, err := db.BeginTx(ctx, pgx.TxOptions{IsoLevel: pgx.ReadCommitted})
  if err != nil { return 0, err }
  defer tx.Rollback(context.Background())

  var runID uuid.UUID
  if err = tx.QueryRow(ctx, 
    `SELECT run_id FROM worker_leases WHERE run_id=$1 FOR UPDATE`,
    l.RunID).Scan(&runID); errors.Is(err, pgx.ErrNoRows) {
    return 0, ErrLeaseLost
  } else if err != nil { return 0, err }

  var valid bool
  err = tx.QueryRow(ctx,
    `SELECT COALESCE(owner=$2 AND epoch=$3 AND expires_at>clock_timestamp(), false)
     FROM worker_leases WHERE run_id=$1`,
    l.RunID, l.Owner, l.Epoch).Scan(&valid)
  if err != nil { return 0, err }
  if !valid { return 0, ErrLeaseLost }

  var revision int64
  err = tx.QueryRow(ctx,
    `UPDATE agent_runs
     SET checkpoint=$3::jsonb, revision=revision+1, updated_at=clock_timestamp()
     WHERE run_id=$1 AND revision=$2 AND status='running'
     RETURNING revision`,
    l.RunID, expectedRevision, string(checkpoint)).Scan(&revision)
  if errors.Is(err, pgx.ErrNoRows) { return 0, ErrConflict }
  if err != nil { return 0, err }
  if err = tx.Commit(ctx); err != nil {
    return 0, fmt.Errorf("commit outcome requires read-back: %w", err)
  }
  return revision, nil
}

Row lock held to transaction end. If B already took over, A's owner/epoch check fails. If A locks the lease row and passes validation, B's takeover waits for A's short transaction to finish. The protocol allows A to write in the narrow window "valid at check, expired at commit" because takeover and commit already have a deterministic serial order; therefore the transaction must be short — strictly no LLM, MCP, HTTP, or queue waits inside.

Two async checkpoints in the same epoch cannot be ordered by epoch alone. Both carry the same expectedRevision; at most one update succeeds. The loser must decide based on step semantics whether to re-read and recompute, not blindly overwrite with a new revision.

07 State Machine, Error Classification, and Heartbeat: Cancellation Reduces Work, Fencing Rejects Stale Commits

State transition table:

Start run : from pending — current lease + revision CAS → running

Save checkpoint : from running — current lease + revision CAS

Normal completion : from running — current lease + revision CAS → succeeded

Deterministic business failure : from running — current lease + revision CAS → failed

User cancel : from pending/running — authorized control-plane state machine → cancelled

A worker that lost its lease must not mark the task failed; it has no write authority. Terminal states cannot return to running; retries must create a new run_id.

Error handling table:

ErrLeaseLost : Confirmed no commit right → Cancel execution; don't write terminal state; preserve known evidence

ErrConflict : Stale snapshot within same epoch → Read back state; decide recompute/stop per step semantics

SQL timeout / connection lost : Result unknown → Don't start next step; read back revision/ledger from primary

40001 (serialization failure): Transaction not committed → Retry whole short transaction from start, with budget limit

Remote timeout : External result unknown → Update ledger to unknown; query or retry with same key

Heartbeat uses a separate timeout context; any unconfirmed renewal cancels the execution context. Cancellation is not remote revocation — the HTTP peer may have already completed the request — so it only reduces subsequent work.

func Heartbeat(ctx context.Context, cancel context.CancelCauseFunc,
  interval, timeout time.Duration, renew func(context.Context) error) {
  ticker := time.NewTicker(interval)
  defer ticker.Stop()
  for {
    select {
    case <-ctx.Done(): return
    case <-ticker.C:
      callCtx, stop := context.WithTimeout(ctx, timeout)
      err := renew(callCtx)
      stop()
      if err != nil {
        cancel(fmt.Errorf("lease renewal unconfirmed: %w", err))
        return
      }
    }
  }
}

TTL=30s, 5s heartbeat, 2s single-shot timeout are only experimental starting points. Parameters should be derived from p99 DB latency, scheduling jitter, observed pause durations, target takeover time, and acceptable false-cancellation rate.

08 Side Effects Are Not Part of the DB Transaction: operation_id and unknown Are the Floor

The most dangerous window:

local intent committed → remote create succeeded → executor crashes → local result not recorded

On retry, cannot infer "remote didn't execute" from "local has no result". Local unique key only deduplicates local registration; it cannot make HTTP atomic. operation_id must express business intent, e.g.: tenant_id / run_id / step_id / effect_version It must never contain worker ID, epoch, retry count, or current timestamp. For LLM dynamic tool calls, persist step plan, request digest, and operation ID in the protected transaction first; the takeover worker resumes the same plan instead of re-inferring a different call identity.

On intent insert, primary-key conflict must compare run_id, tool_name, request_digest: exact match means same intent; otherwise it's a bug or request corruption. Digest uses explicit canonical JSON serialization, not hashing arbitrary JSON text bytes.

09 Executable State Machine for Outbox Claim, Cancel, and Reconciliation

9.1 Register Intent in Same Protected Transaction

When current owner advances checkpoint to "step planned", simultaneously insert tool_operations(status='prepared'). Both commit together, avoiding "state says call tool but intent lost".

9.2 Dispatcher Claims: Only Claim Prepared Ops Still Authorized

Dispatcher is not the task owner, but on claim must confirm run has no terminal/cancelled state, using a unique dispatch owner with a finite deadline. Claim transaction does not span HTTP:

-- $1=dispatcher_owner, $2=claim_ttl_ms
WITH candidate AS (
  SELECT o.operation_id
  FROM tool_operations o
  JOIN agent_runs r ON r.run_id = o.run_id
  WHERE o.status = 'prepared'
    AND r.status IN ('pending','running')
  ORDER BY o.updated_at
  FOR UPDATE OF o SKIP LOCKED
  LIMIT 1
)
UPDATE tool_operations o
SET status = 'dispatching', dispatch_owner = $1,
    dispatch_until = clock_timestamp() + ($2::bigint * interval '1 millisecond'),
    revision = revision + 1, updated_at = clock_timestamp()
FROM candidate c
WHERE o.operation_id = c.operation_id
RETURNING o.*;

Only after successful claim call the remote endpoint, passing the stable operation_id. If dispatcher crashes or times out, do not revert dispatching back to prepared — the remote may have succeeded. Claim expiry only means the local claim session lost contact; subsequent state must transition to unknown first, then query, or retry with the same key for tools that support reliable idempotency.

Reconciler explicitly marks expired claim sessions as unknown in a short transaction; condition retains original dispatch_owner and revision to avoid misprocessing a record just re-claimed or already written back:

-- $1=operation_id, $2=dispatch_owner, $3=expected_revision
UPDATE tool_operations
SET status = 'unknown', dispatch_owner = NULL, dispatch_until = NULL,
    revision = revision + 1, updated_at = clock_timestamp()
WHERE operation_id = $1
  AND status = 'dispatching'
  AND dispatch_owner = $2
  AND revision = $3
  AND dispatch_until <= clock_timestamp();

9.3 Receive Result and Handle unknown

Success response can only be written when expected dispatch_owner, ledger revision, and request digest match. Late responses may be saved as evidence but must not directly mark the run succeeded. The current owner or Reconciler advances the workflow in a protected state transaction based on verified ledger results.

-- $1=operation_id, $2=dispatch_owner, $3=expected_revision,
-- $4=remote_id, $5=result_json
UPDATE tool_operations
SET status = 'succeeded', remote_id = $4, result = $5::jsonb,
    dispatch_until = NULL, revision = revision + 1, updated_at = clock_timestamp()
WHERE operation_id = $1
  AND status = 'dispatching'
  AND dispatch_owner = $2
  AND revision = $3;

Zero rows must not be silently ignored by the caller: it may mean Reconciler already flipped to unknown, record was cancelled, or a stale claim session was used. For tools supporting idempotent query, the response handler can first query ledger/remote by operation_id then decide convergence; never switch to a new operation ID to re-issue create.

Full reconciliation state machine:

prepared + Dispatcher claims → dispatching → Call remote after transaction commits

dispatching + Verified response → succeeded → Save remote_id/result

dispatching + Timeout, process lost → unknown → Save error evidence, do not reset

unknown + Query by operation_id → succeeded / failed_final → Converge to remote fact

unknown + Remote idempotent retry reliable → unknown / succeeded → Retry with original key

unknown + No query, no idempotency → unknown → Alert, manual confirm, or business compensation

prepared + Run cancelled → cancelled → Forbid dispatch

dispatching/unknown + Run cancelled → keep original → Query, compensate, or manual handling

Cancel is not ledger deletion and does not necessarily cancel in-flight side effects. System must explicitly define whether a submitted intent remains authorized to complete; if not, define remote cancel, compensation, or manual handling. Modifying a planned effect increments effect_version, preserving original intent for audit.

Tool capability determines what guarantees can be promised:

Durable idempotent key : Same-key query/retry — Still cannot promise unconfirmed replay outside retention window

Deterministic resource ID + query : Post-create conflict check tag, ownership, spec — Still cannot promise same name = same business intent

Operation query : Query first, then decide next step — Still cannot promise auto-convergence for gaps in acceptance records

Neither : unknown + reconciliation/manual — Still cannot promise automatic, duplicate-free retry

MCP is a calling protocol; it does not automatically give tools idempotent semantics. The adapter layer must explicitly pass the operation ID to the concrete tool, validating its scope, parameter conflict rules, and idempotency retention period.

10 Failover: WAL ≠ Token Never Rolls Back

PostgreSQL default streaming replication is asynchronous; promoting a lagging replica can lose commits already acknowledged on the primary. Therefore "epoch written to WAL" ≠ "epoch never rolls back after any failover".

Runtime contract must specify:

Ownership judged only on the single writable primary; old primary must be fenced before new primary promotion to avoid split-brain.

If safety goal requires preserving acknowledged epochs, deploy synchronous replication, commit acknowledgment, and qualified replica promotion policies to match.

Must not silently degrade to a mode that loses acknowledged writes during failure while still claiming the same guarantee.

After restoring from old backup, first resolve stale workers, external resource high-water marks, ledger, and remote resource history forks before resuming dispatch.

Database server time eliminates client clock skew but cannot prevent database clock jumps. Forward jump causes early takeover; backward jump delays recovery; protected transactions keep state safe; clock monitoring and recovery plans preserve availability.

11 Acceptance Experiments and Verification Matrix

The following must run in isolated environments; do not write expectations as tested conclusions.

A: Continuous checkpoints — Injected fault: Two sequential writes same epoch — Must assert: Both succeed, revision increments consecutively

B: Old owner resumes — Injected fault: A SIGSTOP; lease expires; B takes over; A SIGCONT — Must assert: A's renew, save, release cannot change B's epoch/state

C: Same-epoch reorder — Injected fault: Two requests same expectedRevision concurrent — Must assert: Exactly one succeeds, revision increments by one only

D: Race after validation — Injected fault: A locks lease row then paused by barrier — Must assert: B waits for A's transaction to end before takeover

E: Remote success then crash — Injected fault: Remote create succeeds, kill executor before local result write — Must assert: Idempotent tool: exactly one actual resource; non-idempotent: stays unknown

F: Cancel race — Injected fault: Cancel prepared/dispatching/unknown separately — Must assert: Prepared not dispatched; in-flight items not falsely claimed unexecuted

G: Retry isolation — Injected fault: Late callback from old run arrives at new run — Must assert: Callback cannot advance new run

H: Failover — Injected fault: Controlled promote under async and target sync policies — Must assert: Outcome matches deployment contract; record epoch/ledger/remote high-water

Tests use process-level SIGSTOP/SIGCONT to simulate pause; single-goroutine Sleep does not pause the same-process heartbeat. All barriers, lock waits, and polls must have global timeouts. Performance benchmarks reported separately: claim rate, renewal writes, row-lock waits, WAL, table bloat, autovacuum, pool waits, commit latency.

12 Observability, Capacity, and Permission Boundaries

Metrics/logs: lease_acquire_total{outcome} — Distinguish success, busy, missing, DB error lease_renew_total{outcome} — Distinguish success, lost, result unknown protected_write_rejected_total{reason} — Observe lease, state, revision rejections lease_row_lock_wait_seconds — Detect long transactions blocking renew/takeover tool_operation_unknown_total / oldest age — Find non-converging side effects workflow_takeover_duration_seconds — Measure loss-to-recovery progress

Structured logs with run_id, owner, epoch, revision, operation_id, trace_id — Enable causal tracing

High-cardinality fields ( task_id, run_id, operation_id) go only to logs/traces, not Prometheus labels. With N active runs and heartbeat interval H seconds, steady-state renewal writes ≈ N/H; e.g., 30,000/5 = 6,000/sec is a load estimate, not a DB performance conclusion.

Database roles are also a security boundary. App accounts should not run arbitrary SQL bypassing protected writes; centralize state transitions via stored procedures or controlled services, tighten table-level UPDATE permissions. Ledger request bodies must be masked, encrypted, or stored per data policy.

13 Rollout and Rollback: Consolidate Write Paths First, Then Enable Constraints

Inventory all state write paths, tool side effects, and their authoritative resources.

Introduce run_id, lease, revision, operation ledger, and observability — but do not yet allow mixing old write paths.

Converge checkpoint, terminal states, errors, and intent registration to the protected transaction entry point.

Deploy Workers, Dispatchers, and Reconcilers that understand the new run/operation protocol.

Stop old Workers, drain in-flight requests, revoke their direct table-write permissions, and enforce new constraints.

Complete failure experiments first, then small-traffic validation of takeover, cancel, reconciliation, and recovery drills.

During coexistence, old Workers that still write directly bypass fencing; merely adding fields is not a safety upgrade. Rollback cannot restore a version that can bypass the safety boundary: prioritize stopping dispatch, preserving ledger and external evidence, until a compatible version is restored.

14 Conclusion: The Lock Is Just One Coordinate in the Protocol

For preemptible Agent long tasks, safety comes from a set of jointly holding rules: non-reusable run_id isolates history; current epoch owns state write rights; same epoch ordered by revision and state machine; same side effect keeps stable operation_id; unknown remote results are queryable, reconcilable, or escalated; failover obeys explicit durability contracts.

Redis lock, PostgreSQL lease, fencing token, outbox each solve different problems. Composing them into a clear commit and side-effect protocol ensures that even if an old Worker resumes, it cannot drag the system back into the past.

References

Redis Distributed Locks : random token, lease, and async replication failover window.

PostgreSQL 16 Transaction Isolation : concurrent updates and conditional re-check under READ COMMITTED.

PostgreSQL 16 Explicit Locking : row locks and transaction lifecycle.

PostgreSQL 16 Log-Shipping Standby Servers : durability boundaries of async/sync replication.

AWS Builders' Library: Making retries safe with idempotent APIs : client request identity and safe retries.