Production-Grade Message Bus with Go & NATS JetStream: Transactions, Ordering & Recovery
This article details a production-ready messaging architecture using Go and NATS JetStream, covering transactional outbox publishing, consumer idempotency via inbox tables, aggregate version ordering strategies, quarantine and recovery workflows, Kubernetes R3 deployment, capacity planning, and acceptance testing against failure injection scenarios.
1. Reliability Contract
The author defines a verifiable reliability contract for an order-event system, explicitly stating what is guaranteed and what is not.
Guaranteed
Publish : After business and outbox commit in the same transaction, the relay will eventually attempt to publish the event.
Consume : For the same logical subscription, the specified local database effect is committed at most once per event.
Replay : Within the stream retention window, new consumers can be created to rebuild read models.
Ordering : Duplicate, stale, and gap detection via aggregate version; per-aggregate serialization when required.
Availability : R3 cluster survives a single member failure with majority quorum and can recover service.
Failure handling : Retryable failures, isolated messages, and delivery exhaustion all have durable records and assigned ownership.
Explicitly Not Guaranteed
Infinite capacity for outbox, disk, or cluster.
Exactly-once for external side effects (HTTP, SMS, payments).
Recovery of data expired or deleted by capacity policies.
Worker completion order matching stream sequence.
Automatic operable DLQ from MaxDeliver.
This contract is more engineering-valuable than claiming "end-to-end exactly-once". For external systems, the application must still use the target system's idempotency keys, state queries, and reconciliation mechanisms.
1.1 JetStream vs Kafka: Select by Constraints
JetStream runs inside nats-server and still incurs broker, cluster, disk, Raft, and operational costs. Kafka is not synonymous with "high throughput only". Evaluation must compare at least:
Existing NATS request/reply, subject routing, and service discovery needs.
Dependency on Kafka Connect, Flink, long-term log, data warehouse governance ecosystems.
Event size, retention, replicas, independent consumer count, and replay requirements.
Team operational expertise with NATS/Kafka, disks, backup, and failure drills.
Benchmark results under identical durability, replication, message size, consumer count, and storage conditions.
Do not replace capacity design with a rule-of-thumb QPS threshold (e.g., "under 100k QPS choose NATS"). A 1 KiB vs 1 MiB message, R1 vs R3, one consumer vs 100 independent consumers are different systems.
2. Model: Subject, Stream, Consumer, Worker, Aggregate
Subject : routing name (e.g., orders.paid.v1), not a Kafka partition.
Stream : stores matching subjects, defines storage, replication, retention, capacity limits.
Consumer : logical subscription with filter, delivery, ack progress.
Worker : execution process; multiple instances bind to the same durable pull consumer to share work.
Aggregate : business ordering boundary (e.g., an order), distinct from subject or consumer.
Flow diagram: Order Transaction → Outbox → Relay → ORDERS Stream (R3) → billing-v1 Consumer, notification-v1 Consumer, rebuild-* Consumers → Shared Worker Pool → Inbox + Billing Projection, Isolation Records, Recovery.
Billing and notification each need their own consumer to process once; multiple billing pods share billing-v1. If each pod created a different durable, all would receive the full event stream, defeating load sharing.
2.1 Retention Policy
Three policies compared:
LimitsPolicy : Business events, multi-subscription, rebuild. Ack does not delete; still subject to time/count/byte limits. Chosen for independent billing, notification, and controlled rebuild. Retention window and capacity limits become part of the business recovery window.
WorkQueuePolicy : Single task consumption direction. Ack deletes; cannot overlap independent filters.
InterestPolicy : Retain only while subscribers interested; may remove immediately if no matching consumer.
3. Reliable Publishing: Outbox is Half the Transaction
3.1 Same-Transaction Business Table and Outbox Write
CREATE TABLE event_outbox (
event_id uuid PRIMARY KEY,
aggregate_id text NOT NULL,
aggregate_ver bigint NOT NULL,
subject text NOT NULL,
payload jsonb NOT NULL,
payload_sha256 bytea NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz,
attempts integer NOT NULL DEFAULT 0,
next_attempt_at timestamptz NOT NULL DEFAULT now(),
claim_token uuid,
claim_until timestamptz,
last_error text
);
CREATE INDEX idx_outbox_ready ON event_outbox(next_attempt_at, created_at) WHERE published_at IS NULL;
CREATE UNIQUE INDEX uq_outbox_aggregate_version ON event_outbox(aggregate_id, aggregate_ver);Order state change and outbox insert must commit in the same DB transaction. event_id represents an immutable business fact; retries reuse the same ID. The unique index on (aggregate_id, aggregate_ver) exposes protocol errors where the same aggregate version generates two different events. This index applies only when the version is the unique sequence number for that aggregate's events.
3.2 Relay: Short Lease, Conditional Write-Back, Stable Message ID
Relay state machine:
Short transaction with FOR UPDATE SKIP LOCKED to pick due, unpublished, lease-expired records.
Write new claim_token, claim_until, increment attempts, commit.
Call JetStream publish API outside transaction with Nats-Msg-Id = event_id.
On PubAck, conditionally write published_at using event_id + claim_token.
On failure, log error, compute exponential backoff; process crash lets lease expire for another relay to take over.
ack, err := js.Publish(ctx, record.Subject, record.Payload,
jetstream.WithMsgID(record.EventID.String()))
if err != nil {
// publish result unknown; cannot mark published
return err
}
// record ack.Stream, ack.Sequence, ack.Duplicate for audit and debugging
_ = ackUPDATE event_outbox
SET published_at = now(), claim_token = NULL, claim_until = NULL
WHERE event_id = $1 AND claim_token = $2 AND published_at IS NULL;Crash after publish but before write-back leads to expected duplicate publish path. Marking published before calling the message system would create an unrecoverable loss window. Lease coordinates relays but is not a publish fence; stable event_id, publish deduplication, and consumer idempotence remain essential.
3.3 Deduplication Window vs Business Event Identity
Stream Duplicates window deduplicates by Nats-Msg-Id only within the window; outside, same ID may re-enter stream. Consumer must remain idempotent forever. Never reuse same ID for different payload; store payload digest and treat ID conflict as data protocol incident. Per-subject "infinite deduplication" requires specific subject mapping, retention, and discard conditions; it is not simply setting Duplicates to infinite.
3.4 Relay Ordering: Version Field ≠ Automatic Ordering
Parallel relays may emit version 12 before 11. Including aggregate_version in payload only lets consumer detect the problem, not auto-repair. Three explicit strategies:
Per-aggregate serial relay : Claim one record per aggregate at a time, publish in version order. Low implementation complexity; hot aggregate throughput limited.
Fixed-shard relay : Hash aggregate ID, single shard executes sequentially. Horizontal scaling; requires shard ownership/fencing.
Consumer gap table : On future version, write to waiting area, advance after gap filled. Suitable for integration where source order not guaranteed; needs query and timeout compensation.
"Limit concurrency by pod count" is not fencing. Shard owner needs lease and generation token; DB version condition is final arbiter.
4. Stream & Consumer: Codify Capacity and Behavior
Example using github.com/nats-io/nats.go/jetstream. Production should use IaC or dedicated control plane, not business pods mutating shared resources on startup.
func provision(ctx context.Context, js jetstream.JetStream) error {
stream, err := js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
Name: "ORDERS",
Subjects: []string{"orders.*.v1"},
Storage: jetstream.FileStorage,
Replicas: 3,
Retention: jetstream.LimitsPolicy,
Discard: jetstream.DiscardNew,
MaxAge: 7 * 24 * time.Hour,
MaxBytes: 50 << 30,
MaxMsgs: 20_000_000,
MaxMsgSize: 64 << 10,
Duplicates: 10 * time.Minute,
MaxConsumers: 64,
})
if err != nil { return fmt.Errorf("configure ORDERS: %w", err) }
_, err = stream.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "billing-v1",
FilterSubject: "orders.paid.v1",
DeliverPolicy: jetstream.DeliverAllPolicy,
AckPolicy: jetstream.AckExplicitPolicy,
AckWait: 30 * time.Second,
MaxDeliver: 8,
MaxAckPending: 64,
})
return err
}Key interpretations: DiscardNew rejects publish at capacity; relay must keep outbox unpublished and alert, not swallow error and mark success. MaxAge, MaxBytes, MaxMsgs are jointly effective limits; "retain seven days" is only age limit, not capacity guarantee. MaxAckPending is shared across all workers bound to the consumer, not per pod. MaxDeliver=8 includes first delivery; reaching limit is not auto-DLQ, requires recovery. MaxConsumers must cover resident subscriptions and concurrent rebuilds, but not unlimited; account-level defaults and stream limits must be validated together. MaxMsgSize, MaxAckPending, client prefetch, client cache, single fetch max_bytes are different layers; long-running services prefer SDK continuous consume; Fetch(1) suits minimal processing unit or low-throughput tasks.
5. Consumption Effect: Inbox Unique Key and Business Update Same Transaction
5.1 Data Structures
CREATE TABLE consumer_inbox (
consumer_name text NOT NULL,
event_id uuid NOT NULL,
payload_sha256 bytea NOT NULL,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_name, event_id)
);
CREATE TABLE billing_projection (
order_id text PRIMARY KEY,
paid boolean NOT NULL DEFAULT false,
applied_version bigint NOT NULL DEFAULT 0,
last_processed_event uuid,
updated_at timestamptz NOT NULL DEFAULT now()
);consumer_nameis logical subscription name, not pod name. Same business subscription scaling shares inbox domain; different subscriptions keep separate processing facts.
5.2 Transactional Processing: Duplicate-Safe, Version-Safe, Correct Audit Semantics
func applyPaid(ctx context.Context, db *sql.DB, ev PaidEvent, raw []byte) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil { return err }
defer tx.Rollback()
// Lock aggregate row, check version first. Gap path must not write Inbox; otherwise replay after fill would be incorrectly seen as "already processed". New aggregate treated as appliedVersion=0.
appliedVersion, err := lockAppliedVersion(ctx, tx, ev.OrderID)
if err != nil { return err }
if ev.AggregateVersion > appliedVersion+1 {
if err := recordGap(ctx, tx, ev, appliedVersion); err != nil { return err }
if err := tx.Commit(); err != nil { return err }
return ErrVersionGap // caller NakWithDelay; Inbox not yet written.
}
digest := sha256.Sum256(raw)
result, err := tx.ExecContext(ctx, `
INSERT INTO consumer_inbox(consumer_name, event_id, payload_sha256)
VALUES ('billing-v1', $1, $2)
ON CONFLICT (consumer_name, event_id) DO NOTHING`, ev.EventID, digest[:])
if err != nil { return err }
inserted, err := result.RowsAffected()
if err != nil { return err }
if inserted == 0 {
var stored []byte
err = tx.QueryRowContext(ctx, `
SELECT payload_sha256 FROM consumer_inbox
WHERE consumer_name = 'billing-v1' AND event_id = $1`, ev.EventID).Scan(&stored)
if err != nil { return err }
if !bytes.Equal(stored, digest[:]) {
return fmt.Errorf("event ID reused with a different payload: %s", ev.EventID)
}
return tx.Commit()
}
if ev.AggregateVersion <= appliedVersion {
// Older event doesn't change current projection, but acknowledged processed; future redelivery can take Inbox fast path.
return tx.Commit()
}
_, err = tx.ExecContext(ctx, `
INSERT INTO billing_projection(order_id, paid, applied_version, last_processed_event)
VALUES ($1, true, $2, $3)
ON CONFLICT (order_id) DO UPDATE
SET paid = true,
applied_version = EXCLUDED.applied_version,
last_processed_event = EXCLUDED.last_processed_event,
updated_at = now()
WHERE billing_projection.applied_version = EXCLUDED.applied_version - 1`,
ev.OrderID, ev.AggregateVersion, ev.EventID)
if err != nil { return err }
return tx.Commit()
}lockAppliedVersionuses SELECT ... FOR UPDATE (create/lock initial row if missing per domain); recordGap commits in same transaction. Caller receives ErrVersionGap and does delayed NAK to keep message redeliverable; committed gap record handles compensation, query, or rebuild. Must not write Inbox first for gap.
Three version paths: applied_version >= event.version: duplicate/stale, commit then ack. applied_version + 1 == event.version: normal advance. applied_version + 1 < event.version: version gap, do NOT ack as success. Write persistent aggregate_gap waiting area, trigger query/compensation/rebuild.
If using raw JSON bytes for digest, publish retries must keep identical serialization; if link re-encodes JSON, switch to defined canonical digest. Do not periodically truncate Inbox; retention must cover online redelivery, manual redelivery, historical rebuild, and backup-restore duplicate ranges.
5.3 External Side Effects Need Separate State Machine
DB unique key cannot replace third-party payment/email/SMS idempotency protocols. Recommended local effect_outbox: store business idempotency key, request digest, attempt count, remote status, reconciliation timestamp; on timeout, query or reconcile first, then decide retry.
6. Consume Loop: Bounded Concurrency, Protocol Ack, Graceful Shutdown
Worker order: validate → local transaction → ack. If transaction succeeds but ack result unknown, allow JetStream redelivery and rely on Inbox to detect duplicate; do not redo committed external side effects.
func handle(ctx context.Context, msg jetstream.Msg, db *sql.DB,
quarantine func(context.Context, jetstream.Msg, error) error) {
taskCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
ev, err := validatePaid(msg.Data())
if err != nil {
if err := quarantine(taskCtx, msg, err); err != nil {
_ = msg.NakWithDelay(15 * time.Second)
return
}
// Isolation persisted; Term failure may cause redelivery, isolation side must deduplicate by source triple.
_ = msg.Term()
return
}
if err := applyPaid(taskCtx, db, ev, msg.Data()); err != nil {
_ = msg.NakWithDelay(15 * time.Second)
return
}
ackCtx, ackCancel := context.WithTimeout(ctx, 2*time.Second)
defer ackCancel()
if err := msg.DoubleAck(ackCtx); err != nil {
slog.Warn("ack result uncertain; inbox will absorb redelivery", "event_id", ev.EventID, "err", err)
}
}DoubleAckreduces ack uncertainty but cannot make DB commit and ack a distributed atomic transaction; its correct use is shrinking uncertainty window, not claiming exactly-once.
Concurrency first constrained by DB budget. With input rate λ, avg processing time E[T], target utilization u, initial slots: C_needed ≈ λ × E[T] / u. Must also verify tail latency, connection pool, lock contention, retry storms, and MaxAckPending. Suggest satisfying:
T_localQueue + T_businessBudget + T_ackBudget + T_margin < AckWait. Long tasks may periodically call InProgress() to extend ack wait, but it's not a business lock; network partition or GC pause may still cause old/new worker parallelism, ultimately resolved by version condition, DB lock, or third-party idempotency.
6.1 SIGTERM Stop Sequence
Set readiness unavailable; cancel pull-only stopPull context.
Stop initiating new pulls; already fetched messages continue processing with separate workCtx.
Wait for workers within business grace period; only then cancel workCtx.
After workers finish, close NATS connection and DB pool. FetchMaxWait counts in shutdown budget. Do not close NATS connection before workers attempt ack.
7. Quarantine, Delivery Exhaustion, Recovery: Failures Must Have Ownership
7.1 Quarantine Record Persisted Before Termination
Quarantine table/stream stores: source stream, source sequence, consumer, original subject, original payload, event ID (if parsable), error classification, delivery count, first/last failure time. Recommended unique key: PRIMARY KEY (source_stream, source_sequence, consumer_name). This makes "quarantine succeeded, Term() before crash" redelivery safely deduplicatable. Quarantine subject outside original stream match domain, e.g., quarantine.orders.paid.v1, to avoid reflow.
Failure type → Action:
JSON corrupt, schema unsupported → Quarantine success then Term().
DB/dependency transient fault → NakWithDelay(); global fault pause or rate-limit pull.
Inbox hit with matching digest → Ack, normal duplicate.
Same ID different content → High-priority quarantine, alert, freeze problematic producer.
Ack result unknown → Record uncertainty; wait redelivery, no duplicate business effect.
Reach MaxDeliver → Recovery takes over; not automatic DLQ. BackOff affects ack-timeout redelivery rhythm; plain Nak() does not auto-adopt it. Need delay explicitly with NakWithDelay(), verify interaction with locked version.
7.2 Recovery is a Stateful Product Component
Recovery cannot just listen to advisories. Needs persistent task table and periodic reconciliation:
Input: MaxDeliver advisory → Persistent action: Upsert recovery task by (stream, sequence, consumer) → Completion: Original message quarantined, redelivered, or explicitly expired.
Input: Quarantine record → Persistent action: Create manual/auto-fix task → Completion: Operation log, reason, operator, result complete.
Input: Version gap → Persistent action: Write aggregate_gap, query source or start rebuild → Completion: Missing version filled or failure escalated.
Input: Periodic reconciliation → Persistent action: Compare business state, inbox, quarantine, outbox, retention window → Completion: Discover orphaned failures or unrecoverable data.
Recovery task deadline must be less than remaining readable retention of original stream. If original message already expired, do not fail silently: mark unrecoverable, escalate incident, execute补数 from authoritative business DB or backup.
Manual redelivery must audit operator, reason, source, payload digest. Unchanged business fact keeps original event_id; corrected new fact generates new ID with original_event_id recorded. Publish deduplication ID and business event ID may differ, but Inbox always judges completion by business event identity.
7.3 Rebuild Consumer Lifecycle
Rebuild is not "create a permanent consumer". Each rebuild-* task must register: owner, purpose, start point, target table, deadline, max read rate, deletion acceptance. After completion, verify target data, then delete consumer; on failure, pause, retain diagnostics, owner decides continue or cleanup. Count rebuild consumers in MaxConsumers and account resource budget to avoid temporary tasks crowding resident subscriptions.
8. Kubernetes: R3, Persistent Volumes, Failure Domains, Backup Verified Separately
R3 minimum conditions:
Three NATS nodes form communicable cluster with stable identities.
Each node enables JetStream with independent persistent volume.
Target stream configured Replicas=3, replica sync healthy.
Pods and storage spread across expected failure domains to avoid single node/zone/storage failure taking majority.
config:
cluster:
enabled: true
replicas: 3
jetstream:
enabled: true
fileStore:
enabled: true
dir: /data
pvc:
enabled: true
size: 100GiSnippet aligns with current official Chart direction, but delivery must run helm show values and helm template with locked Chart version, review generated StatefulSet, PVC, probes, PDB, NATS config. Also configure storage class, resource requests/limits, TLS, auth, topology spread/anti-affinity, PDB, rolling upgrade strategy, monitoring network isolation. minAvailable: 2 only reduces voluntary eviction risk, cannot withstand node failure.
8.1 fsync and PubAck Boundary
File Storage + PubAck ≠ every message on physical media. sync_interval: always increases sync write strength but significantly impacts throughput/latency; still no substitute for replicas, failure domains, backup, and recovery drills.
jetstream {
store_dir: /data/jetstream
sync_interval: always
}Choose sync strategy per RPO, validate with controlled power loss or equivalent fault, not just kill -9. Upgrade one node at a time: wait for rejoin, target stream replica sync, publish/consume probes recover, then next node.
PVC and R3 are not backups. Accidental delete, wrong retention, wrong credentials, logical corruption replicate to replicas; business DB, inbox, outbox, NATS config, JetStream data all need independent backup and isolated recovery drills.
9. Capacity, Backpressure, Observability: Connect Metrics to Business Consequences
Logical payload space estimate: S_logical = λ × B_avg × T_retention. Example: 1000/s, 1 KiB avg, 7 days → ~577 GiB; R3 payload replicas ~1.69 TiB, not counting index, file overhead, consumer state, recovery headroom, disk watermark. Hence MaxBytes=50 GiB and "full 7-day retention" cannot both hold.
Backlog Q drains only when μ > λ: T_drain ≈ Q / (μ - λ).
Must-observe semantics by layer:
Relay / Publisher : PubAck latency, timeout, rejection, unknown result, oldest unpublished outbox age.
Stream : Message count, bytes, oldest message, leader, replica sync, disk watermark.
Consumer : Pending, ack pending, redelivery, ack floor, delivery exhaustion.
Business App : End-to-end event age, transaction duration, idempotency hits, quarantine count, version gaps.
Recovery : Unclosed recovery tasks, remaining retention time, manual handling duration, unrecoverable count.
Infrastructure : Disk latency/space, CPU throttling, network errors, DB pool and lock waits.
During global DB failure, don't just scale workers or immediate Nak. Pause/rate-limit pull, throttle ingress and relay, preserve recovery space — safer than amplifying retry storm.
10. Security & Governance
TLS and separate credentials; separate permissions for publish, consume, resource management, backup, raw message read.
Place 8222 monitoring port in controlled network; don't assume it inherits client port auth semantics.
No PII (phone, email, tokens) in subjects; quarantine data same retention and access control.
Maintain event catalog: producers, consumers, schema, compatibility rules, max size, retention, failure owner, delete/rebuild conditions.
Define backward/forward compatibility for schema versions; unsupported versions go to quarantine, not business success ack.
11. Production Acceptance: Prove by Failure Windows, Not Just Send QPS
At minimum in three-node R3, PostgreSQL, Relay, two shared durable workers, and recovery environment, complete experiments:
Pre-publish crash : Inject exit after business+outbox commit → Relay recovers and eventually publishes.
PubAck uncertainty : Block PubAck return → Outbox not marked success; stable ID retry; local effect once.
Commit-before-ack crash : Exit between commit and ack → Redelivery occurs; Inbox hit; no duplicate business effect.
Outbox reorder : Same order adjacent versions processed by different relays in parallel → Selected ordering strategy correctly advances or forms recoverable gap.
Transaction rollback : Inject business SQL failure → Inbox and business effect both uncommitted, retryable later.
Poison message : Corrupt JSON / unsupported schema → Quarantine complete, no loop, auditable.
Delivery exhaustion : Continuous retryable failures → Advisory/reconciliation creates recovery task, message not unowned.
Recovery downtime : Recovery down covering delivery exhaustion window → Post-recovery scan/reconciliation rebuilds tasks, within retention.
Single node failure : Stop one NATS node → Eventually majority and replica sync recover, verify sequence and business results.
Majority loss : Stop two members → Write unavailable as expected, business does not falsely report success.
Capacity exhaustion : Lower test stream MaxBytes → Publish rejection observable; outbox retained; backpressure effective.
Graceful shutdown : SIGTERM during transaction processing → Stop pull, time-bound completion, then close connections.
Backup restore : Restore to isolated cluster → Verify events, consumers, business DB, and duplicate handling boundaries.
Load test report must record server/SDK/Chart versions, image digests, CPU, storage, message size distribution, replicas, sync strategy, consumer count, concurrency, DB logic, and percentile latencies during fault phases. "Send QPS" alone cannot prove business completion.
12. Recommended Implementation Order
Define event identity, schema, aggregate version; complete single-node outbox, inbox, post-transaction ack.
Finish quarantine, recovery, version gap handling, manual redelivery audit, graceful shutdown.
Choose and implement per-aggregate ordering strategy; complete reorder and gap fault injection.
Deploy R3, verify failure domains, persistent volumes, single-node upgrade, majority loss, backup restore.
Calibrate retention, disk, MaxAckPending, consumer budget, backpressure, alerts with real load.
Pilot low-risk events first; if dual-write or Kafka bridge, unify business event identity, and implement inbox, outbox, loop control, recovery for the bridge itself.
Conclusion
JetStream's value is not eliminating distributed failure windows for the business, but providing sufficient persistence, replication, and consumption control to make those windows explicitly designable, observable, and recoverable. A production-grade loop closes: outbox prevents loss, stable event ID enables retry, inbox prevents local duplicate effects, version strategy handles reorder, quarantine and recovery own failures, retention and backup define recovery boundaries. Only after these loops are complete does the team truly own an explainable, verifiable, operable message bus.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
