Beyond Handlers: Reliable Order Services with Kratos — Layering, Idempotency, Consistency & Capacity Governance

Introduction: Why Orders Still Duplicate After Switching Frameworks

An order endpoint typically starts with a few steps: validate params, query product, reserve inventory, write order, emit message. As traffic and dependencies grow, these steps amplify each other in failure windows: inventory RPC timeout triggers retry, causing double reservation; order transaction commits but process crashes before Kafka send; slow database exhausts connection pool; rolling deployments route requests to terminating instances. These problems don't disappear by switching from Gin to Kratos. Frameworks provide contracts, transport adapters, and lifecycle tools; reliability must still be implemented via business semantics, transaction boundaries, message protocols, and recovery flows.

The article answers four verifiable questions:

Same business intent retried 100 times — why only one order and one inventory effect?

Process killed at any gap between DB commit, Kafka ack, consumer offset commit — why does system still converge?

Cancellation, reservation success, and payment notification arrive out of order — why no double reservation or lost payment?

During scale-out, slow dependencies, rolling deploy — why does system protect itself instead of amplifying failure?

1. Define "Success": What the Interface Actually Returns

1.1 Order Creation Semantics

Adopts "persist order intent first, then async reserve inventory" pattern: CreateOrder succeeds returning order_id, stable reservation_id, and PENDING_STOCK.

Order, idempotency mapping, and StockReservationRequested Outbox event committed in same local transaction in order DB.

Inventory service reserves successfully → order becomes WAIT_PAY; insufficient → REJECTED.

Creation success ≠ inventory locked ≠ payment done. Client shows "order processing", polls/subscribes for final state.

This is not degraded eventual consistency but an async workflow with explicit promises. If product contract requires "HTTP 201 means inventory locked", use sync reservation with recoverable orchestration or wait for async result within deadline; don't silently change success semantics.

1.2 Five Immutable Constraints

Invariant: Same tenant, user, idempotency key → at most one order. Enforced by: Order table unique index. Acceptance test: Concurrent 100× same key.

Invariant: Same idempotency key cannot represent different business intent. Enforced by: Normalized request fingerprint. Acceptance test: Replay after changing SKU/qty.

Invariant: Committed order intent must have recoverable inventory command. Enforced by: Order + Outbox same transaction. Acceptance test: Kill process immediately after commit.

Invariant: Same reservation_id → at most one inventory reservation effect. Enforced by: Inventory reservation unique index + local transaction. Acceptance test: Duplicate consumption & redelivery.

Invariant: State transitions only along allowed edges. Enforced by: Conditional updates, version numbers, idempotent event handling. Acceptance test: Duplicate, out-of-order, concurrent events.

"Exactly once" across MySQL, Kafka, payment channels is not a blanket promise. Relies on at-least-once delivery, receiver idempotency, conditional state updates, persisted compensation, and reconciliation to make business effects recoverable and convergent.

1.3 Three Stable Identifiers

idempotency_key : Client generates per creation intent; reused on timeout retry. Purpose: Prevent duplicate order creation.

order_id : Order service generates before first creation transaction. Purpose: Order aggregate primary identifier.

reservation_id : Order service generates before first creation transaction; usually one per order. Purpose: Stable idempotency key for inventory reserve, release, confirm. reservation_id must not be regenerated per delivery, nor generated ad-hoc by inventory service. Order service uses same value when writing order, Outbox, and subsequent cancel commands; inventory service uses it as unique key. Thus even if Kafka redelivers in "acked but not yet marked SENT" window, inventory produces only one reservation effect.

2. Contracts: What Client Decides, What Server Promises

2.1 Proto Is Not Just DTO

syntax = "proto3";
package api.order.v1;
option go_package = "example.com/order/api/order/v1;v1";
import "google/api/annotations.proto";

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderReply) {
    option (google.api.http) = { post: "/v1/orders" body: "*" };
  }
  rpc GetOrder(GetOrderRequest) returns (GetOrderReply) {
    option (google.api.http) = { get: "/v1/orders/{order_id}" };
  }
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING_STOCK = 1;
  ORDER_STATUS_WAIT_PAY = 2;
  ORDER_STATUS_REJECTED = 3;
  ORDER_STATUS_CANCELLING = 4;
  ORDER_STATUS_CANCELLED = 5;
  ORDER_STATUS_PAID = 6;
  ORDER_STATUS_REFUNDING = 7;
  ORDER_STATUS_REFUNDED = 8;
}

message CreateOrderRequest {
  string sku_id = 1;
  int32 quantity = 2;
  string idempotency_key = 3;
  string quote_id = 4; // if short-term price lock, client sends server-issued quote_id; no amount
}

message CreateOrderReply {
  string order_id = 1;
  string reservation_id = 2;
  OrderStatus status = 3;
  bool idempotent_replay = 4;
}

message GetOrderRequest { string order_id = 1; }

message GetOrderReply {
  string order_id = 1;
  OrderStatus status = 2;
  string rejection_reason = 3;
  int64 total_price_minor = 4;
  string currency = 5;
}

Request deliberately omits user_id, tenant_id, amount: identity from trusted auth context, price from server quote or product snapshot. For proxy ordering, establish separate auth rules instead of trusting request body user ID. Amount uses minor currency integer with currency code; quantity must be >0, have business upper bound, and multiplication overflow check.

Idempotency key can be in agreed Header (mapped to command at transport layer) or in body; but must limit length, forbid full logging, and ensure SDK reuses original value on retry. Key is semantic stability, clear boundary, not accidentally replaced by gateway retry.

2.2 Errors Are Part of Caller Recovery Protocol

INVALID_ARGUMENT (400 / InvalidArgument) → Correct input

UNAUTHENTICATED (401 / Unauthenticated) → Refresh identity then retry

IDEMPOTENCY_CONFLICT (409 / AlreadyExists) → Change business intent or replay original request

ORDER_NOT_FOUND (404 / NotFound) → Verify order ID and access permission

RESOURCE_EXHAUSTED (429 / ResourceExhausted) → Backoff retry same idempotency key

TEMPORARILY_UNAVAILABLE (503 / Unavailable) → Limited retry or query order

OUTCOME_UNKNOWN (503 / Unavailable with recoverable reason) → Query or replay with same idempotency key

Request timeout only means caller didn't receive result, does not mean transaction necessarily uncommitted . Especially when Commit returns error, result may be uncertain: server must return recoverable error, caller must query or replay same idempotency key, not create new key and reorder.

Error encoder, client SDK, and contract tests must jointly solidify above mapping. Don't assume any framework auto-maps business reason to desired HTTP status.

2.3 State Query & Notification Are Part of Async Semantics

If CreateOrder returns PENDING_STOCK, GetOrder is not optional but part of recovery path: client relies on it after create timeout, network loss, push loss, page restart. Notification via Webhook, WebSocket, polling, or message subscription must satisfy:

Query is always authoritative; notification only reduces wait time.

Immediate post-create query must hit primary or explicit read-your-writes path.

Notification carries order_id, status, status version; no payment tokens or full auth headers.

Client deduplicates by status version; out-of-order notifications cannot overwrite newer state.

3. Layering: Focus on Compile-Time Dependency Direction

3.1 Runtime Call Order ≠ Source Import Direction

service : DTO conversion, identity extraction, protocol error translation. Must not leak: SQL, transactions, inventory retries.

biz : Creation semantics, state rules, permissions, domain interfaces. Must not leak: HTTP headers, ORM models.

data : Atomic persistence, cache, external RPC adapters. Must not leak: External HTTP status codes.

server : Middleware, listeners, route registration. Must not leak: Order state branches.

worker : Outbox, compensation, reconciliation execution loops. Must not leak: Protocol DTOs.

cmd : Object assembly, startup, shutdown order. Must not leak: Single-order business details.

Framework won't auto-prevent biz importing data; need module boundary checks, code reviews, tests. Small projects can use a coarse-grained repo method for full creation transaction; don't split atomic ops into multiple auto-commits just for tidy directories.

3.2 Recommended Directory Structure

api/order/v1/          Proto, error defs, generated code
cmd/order/             main, object assembly, resource shutdown
internal/server/       HTTP, gRPC, admin ports
internal/service/      Protocol adapters
internal/biz/          Use cases, commands, state machine, domain interfaces
internal/data/         MySQL, cache, Kafka/inventory client implementations
internal/worker/       Outbox, compensation, reconciliation
migrations/            Schema & index migrations
configs/               Non-sensitive config samples

biz

doesn't accept Proto Request nor return ORM Model. Cost of copying few DTOs usually lower than forcing transport, business, persistence to share one unevolvable big struct.

3.3 Wire Is Just Assembly Tool

Constructor injection makes object graph visible; hand-written assembly or code gen both fine. Google Wire archived; existing projects can lock tool version, commit generated files, and run git diff --exit-code in CI after regeneration. Regardless of assembly method, initialization failure after connection pools, producers, exporters created must release them; object graph constructible ≠ inventory idempotency or shutdown order correct.

4. Local Transaction: One Create Commits Three Fact Types

4.1 Minimal Data Model (MySQL 8.0, InnoDB)

CREATE TABLE orders (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  order_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  reservation_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  tenant_id BIGINT NOT NULL,
  user_id BIGINT NOT NULL,
  idempotency_key VARCHAR(128) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  request_hash BINARY(32) NOT NULL,
  request_hash_version SMALLINT NOT NULL,
  sku_id VARCHAR(64) NOT NULL,
  quantity INT NOT NULL,
  unit_price_minor BIGINT NOT NULL,
  currency CHAR(3) CHARACTER SET ascii NOT NULL,
  status VARCHAR(32) CHARACTER SET ascii NOT NULL,
  version BIGINT NOT NULL DEFAULT 0,
  created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
  PRIMARY KEY (id),
  UNIQUE KEY uk_order_id (order_id),
  UNIQUE KEY uk_reservation_id (reservation_id),
  UNIQUE KEY uk_create_key (tenant_id, user_id, idempotency_key),
  KEY idx_user_orders (tenant_id, user_id, created_at, id),
  CONSTRAINT chk_quantity CHECK (quantity > 0),
  CONSTRAINT chk_price CHECK (unit_price_minor >= 0)
) ENGINE=InnoDB;

CREATE TABLE order_outbox (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  event_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  aggregate_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  aggregate_version BIGINT NOT NULL,
  event_type VARCHAR(64) CHARACTER SET ascii NOT NULL,
  schema_version SMALLINT NOT NULL,
  payload JSON NOT NULL,
  status VARCHAR(16) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT 'PENDING',
  attempts INT NOT NULL DEFAULT 0,
  next_attempt_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  lease_until DATETIME(6) NULL,
  lease_token VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NULL,
  last_error_code VARCHAR(64) NULL,
  last_error_at DATETIME(6) NULL,
  failed_at DATETIME(6) NULL,
  created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  sent_at DATETIME(6) NULL,
  PRIMARY KEY (id),
  UNIQUE KEY uk_event_id (event_id),
  KEY idx_pending (status, next_attempt_at, id),
  KEY idx_reclaim (status, lease_until, id),
  KEY idx_aggregate (aggregate_id, aggregate_version, id)
) ENGINE=InnoDB;

Idempotency key scoped by tenant and user — a business decision; if B2B uses purchase order number as idempotency scope, index adjusts accordingly. Don't put byte-exact idempotency keys into case-insensitive default collation.

4.2 Request Fingerprint: Only Describes Stable Business Intent

Fingerprint after normalization includes tenant, user, SKU, quantity, quote identifier, and other fields affecting order semantics; records algorithm version. Don't hash arbitrary map serialization; don't include trace ID, request time, or dynamic current price.

After price change, replaying same idempotency key should return first order with its price snapshot, not "parameter conflict". Thus quote identifier can belong to fingerprint; price derived from quote stored as order snapshot, not recalculated on each replay.

Recommended flow: first read primary by idempotency key; if hit, compare fingerprint and return; if miss, validate quote and enter transaction. Pre-query only reduces work; final concurrency correctness still guaranteed by unique index.

4.3 Use Case Depends on One Explicit Atomic Commit Capability

// internal/biz/order.go
type CreateCommand struct {
  TenantID          int64
  UserID            int64
  SKU               string
  Quantity          int32
  IdempotencyKey    string
  RequestHash       [32]byte
  RequestHashVersion int16
  QuoteID           string
}

type Order struct {
  ID              string
  ReservationID   string
  Status          Status
  RequestHash     [32]byte
}

type CreateIntentStore interface {
  // First call generates order_id & reservation_id, writes order and Outbox in same tx.
  // On idempotency conflict, reads existing order from primary, compares fingerprint, no overwrite.
  CreateIntent(ctx context.Context, cmd CreateCommand) (order *Order, replay bool, err error)
}

Repo implementation transaction steps:

Before transaction, generate order_id, reservation_id, event_id. If unique key conflict turns into idempotency hit, these temp values discarded.

With single *sql.Tx, insert order with status PENDING_STOCK and price snapshot.

Same transaction inserts StockReservationRequested Outbox; payload includes event_id, order_id, reservation_id, SKU, quantity, aggregate version, schema version.

Only after commit return creation result to caller.

On unique key conflict, rollback first, then identify conflicting index. Only uk_create_key can be idempotency hit: read existing order from primary, compare request fingerprint, same → return original order, different → return IDEMPOTENCY_CONFLICT.

Cannot treat all duplicate keys as idempotency hits: order_id, reservation_id, event_id collisions are generator or data faults. Don't use INSERT IGNORE to hide errors, nor upsert to overwrite existing fingerprint.

4.4 Don't Wait Cross-Service RPC Inside Transaction

Wrapping inventory RPC inside order DB transaction only turns remote latency into local lock hold time; RPC success but local tx rollback still possible, no cross-DB atomicity. Cross-service: commit order intent + Outbox, then recoverable Worker delivers.

If inventory and order still in same DB, same deployment unit, conditional decrement and order write can be in one local transaction. Service split driven by independent scaling and org boundaries, not to manufacture distributed transactions for microservice showcase.

5. Outbox: How Events Continue After Commit

5.1 Why Handler Goroutine Is Unreliable

context.Background()

emitting events leaves work unmanaged by request or app lifecycle; switching to request ctx also unreliable because HTTP handler return cancels request Context. Bounded channel limits memory and concurrency but cannot recover across process restarts; only suitable for losable telemetry or rebuildable cache tasks.

Critical events must persist first, then delivered by Worker with independent lifecycle.

[Diagram: Order Interface → same-tx write order, reservation_id, event → commit success return PENDING_STOCK → Outbox Worker short-tx claim event & lease → send & wait broker ack → ack → conditional update SENT → at-least-once delivery → consumer inbox, reservation, result Outbox same tx]

5.2 Claim, Lease & Duplicate Window

MySQL 8.0 short transaction with FOR UPDATE SKIP LOCKED to claim queue:

SELECT id, event_id, aggregate_id, aggregate_version, payload
FROM order_outbox
WHERE status = 'PENDING' AND next_attempt_at <= NOW(6)
ORDER BY next_attempt_at, id
LIMIT 100
FOR UPDATE SKIP LOCKED;

During lock hold only flip record to PROCESSING, write random lease_token and lease_until, then commit immediately; Kafka network send outside transaction. On send success, conditional update by id + status + lease_token to SENT; on failure, same condition record error, increment attempts, backoff with jitter back to PENDING. Expired PROCESSING leases reclaimed back to PENDING. lease_token prevents expired Worker overwriting new claim result, but cannot revoke message already sent to Kafka. Therefore "Kafka acked, DB not yet marked SENT" inevitably allows redelivery; consumer must be idempotent on stable event_id and reservation_id.

Multiple Workers don't guarantee per-order event order. When order needed, events carry aggregate version, Kafka partitioned by order_id, limit concurrent in-flight events per aggregate; consumer still must detect stale and out-of-order versions.

5.3 Outbox Must Have Failure Terminal State & Manual Recovery Protocol

"Just keep retrying" lets permanent poison messages consume capacity forever; "dump to DLQ" just moves to another unwatched place. Recommended state machine:

PENDING : Deliverable. Auto: Worker claims. Manual: Can pause or replay.

PROCESSING : Claimed, lease valid. Auto: Success → SENT; recoverable error → PENDING. Manual: Lease anomalies handled by reclaimer.

SENT : Broker acked. Auto: No more auto send. Manual: Only manual replay after audit.

FAILED : Hit retry limit or deemed unrecoverable. Auto: Stop auto deliver, alert. Manual: Fix root cause, create audited replay task.

Failure threshold not just attempts; combine error category, oldest event age, order type, business risk. Replay should not directly modify original SENT record; safer to record operator, reason, linked events, then create new replay event with new event_id, preserving original causal chain.

Must monitor pending count, oldest event age, lease expiry, failure rate, FAILED count, recovery duration. Order compensation scan must correlate long-term PENDING_STOCK with Outbox, inventory reservation, consumer results to find "no longer progressing" orders.

5.4 Consumer Ack After Business Transaction

Inventory consumer local transaction must complete: write inbox dedup record, read/create reservation, conditional decrement inventory, write reservation result, insert StockReserved or StockRejected Outbox. Only after tx commit acknowledge Kafka offset; must not commit offset before business tx succeeds.

Consumer inbox table unique on event_id. Duplicate event handling not "insert fails → error" but read existing result and quietly return; only first processing changes inventory. Consumer process crashes after "business tx committed, offset not committed" → re-consumes but no second business effect.

6. Inventory Reservation: Reserve, Cancel, Release Must All Be Idempotent Together

6.1 Inventory Side Suggested Table & Transaction Order

CREATE TABLE inventory_reservations (
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  reservation_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  order_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
  sku_id VARCHAR(64) NOT NULL,
  quantity INT NOT NULL,
  request_hash BINARY(32) NOT NULL,
  status VARCHAR(16) NOT NULL,
  cancel_requested_at DATETIME(6) NULL,
  created_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  updated_at DATETIME(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
  PRIMARY KEY (id),
  UNIQUE KEY uk_reservation_id (reservation_id),
  KEY idx_order_id (order_id),
  CONSTRAINT chk_reservation_quantity CHECK (quantity > 0)
) ENGINE=InnoDB;

First ReserveStock(reservation_id) transaction order:

Attempt insert reservation record status RESERVING; unique key conflict → read existing, verify order, SKU, quantity, fingerprint, return existing result or report protocol conflict.

For first insert, execute conditional decrement:

UPDATE inventory
SET available = available - ?, reserved = reserved + ?
WHERE sku_id = ? AND available >= ?;

Affected rows = 1 → set reservation to RESERVED; = 0 → REJECTED. Quantity validated before SQL, no negative or zero.

Write StockReserved or StockRejected result Outbox, same transaction as reservation and inventory change.

If cancel command arrives first, inventory service must persist cancel intent: create CANCELLED or CANCEL_REQUESTED placeholder for non-existent reservation; later arriving reserve command sees that state and must not decrement. If reserve already succeeded, cancel executes release once. Cannot rely solely on order service current state because messages arrive out of order.

6.2 Order & Inventory State Machines Designed Together

[State diagram: reservation success & order valid → inventory insufficient → user cancel/timeout → payment confirm → close order → release/cancel confirm → valid payment received → refund confirm → PENDING_STOCK → WAIT_PAY → REJECTED → CANCELLING → PAID → CANCELLED → REFUNDING → REFUNDED]

Inventory side needs at least RESERVED, RELEASED, CONFIRMED, REJECTED, CANCELLED. Order enters CANCELLING then receives late reserve success → must not revert to WAIT_PAY; ensure release command persisted, continue converging to cancel complete.

Order state updates must carry expected state or version:

UPDATE orders
SET status = 'WAIT_PAY', version = version + 1
WHERE order_id = ? AND status = 'PENDING_STOCK' AND version = ?;

Zero rows updated → distinguish duplicate event, late success after cancel, version contention, illegal transition. First two usually idempotent success or trigger compensation; latter two need logging and strategy handling; cannot all be treated as success nor infinite retry.

6.3 Payment Success vs Cancel Race Not Decided by Arrival Order Alone

Payment notification must verify signature, check merchant, currency, amount, order binding, deduplicate by channel transaction ID. Payment success and close-order concurrent must have predefined business policy: allow order recovery, or enter refund + inventory compensation; cannot simply discard "current state disallows" real payment.

Article focuses on creation flow. Full payment state machine, channel query, fund reconciliation designed independently; cannot derive payment success from order.created or inventory reserve result.

7. Context, Timeouts & Retries: Account Waiting in Budget

7.1 Three Work Types Use Different Lifecycles

Sync requests, queries : Context derived from request ctx. Cancel & recovery: Stop waiting on disconnect or deadline.

Outbox, compensation, reconciliation : Context derived from app Worker ctx. Cancel & recovery: Shutdown stops claiming; unfinished tasks recover via persistence.

Exporter flush, resource close : Independent shutdown ctx with deadline. Cancel & recovery: Don't reuse cancelled request or Worker ctx. context.WithoutCancel only cuts cancel propagation, cannot recover work after process restart; not a reliable queue.

7.2 Timeout Is End-to-End Budget

If entry budget 800ms, auth, param validation, quote, DB connection wait, transaction, RPC, encode response all share it; cannot let each layer get fresh 800ms. Downstream deadline = min(configured cap, parent ctx remaining budget minus return margin).

DB connection acquisition, admission queue, DNS, LB wait also consume budget. Monitoring only SQL execution misses tail latency from pool wait.

7.3 Retry Must Have Single Responsible Layer

Write requests retry limited times only when idempotency constraints hold: create order reuses idempotency_key, inventory command reuses reservation_id. If gateway, Service Mesh, Kratos client, and biz layer each retry 3×, one call amplifies to dozens.

Define retry responsible layer, total budget, max attempts, jitter backoff, observable attempts. Param errors, insufficient inventory, idempotency conflict, permission failures must not enter infra retry.

8. Kratos Middleware: Types, Positions, Metrics Scope

8.1 Different Interceptors Cannot Mix Directly

Kratos middleware.Middleware differs from grpc-go UnaryServerInterceptor, StreamServerInterceptor. Native Sentinel gRPC adapter cannot be shoved into kgrpc.Middleware(...). Snippet shows Kratos v2 wiring positions; adapter construction must match project-locked versions:

import (
  "github.com/go-kratos/kratos/v2/middleware"
  kgrpc "github.com/go-kratos/kratos/v2/transport/grpc"
  ggrpc "google.golang.org/grpc"
)

func NewServer(
  chain []middleware.Middleware,
  sentinelUnary ggrpc.UnaryServerInterceptor,
  sentinelStream ggrpc.StreamServerInterceptor,
) *kgrpc.Server {
  return kgrpc.NewServer(
    kgrpc.Address(":9000"),
    kgrpc.Middleware(chain...),
    kgrpc.UnaryInterceptor(sentinelUnary),
    kgrpc.StreamInterceptor(sentinelStream),
  )
}

Unary middleware doesn't auto-cover streaming; gRPC config doesn't auto-protect HTTP. HTTP needs corresponding middleware or shared Kratos rate-limiter.

8.2 Recovery Order Depends on What You Want to Observe

Common conceptual order: tracing → metrics/access log → recovery → auth → param validation → admission → business. Outer logs/metrics see recovery-transformed errors; cost: inner recovery can't catch panic from outer components. Add outermost fallback if needed, avoid duplicate recording.

Admin endpoints registered directly on HTTP router, decode failures, unmatched routes may bypass business middleware. Tests must cover panic, auth failure, param decode failure, rate limit, stream cancel; verify error codes, logs, metrics on each chain.

8.3 Rate Limit, Concurrency Guard, Circuit Breaker Are Not Same Thing

Per-user/tenant quota prevents single principal exhausting resources, but doesn't replace global admission.

Local concurrency guard limits in-flight work; suitable when dependency slows to prevent queue buildup.

QPS rate limit restricts arrival rate; must specify per-instance or global.

Downstream circuit breaker isolates faulty dependency; needs min sample size, stats window, error classification, half-open probe cap.

Insufficient inventory and idempotency conflict should not count as dependency failure. Thresholds derived from SLO, dependency patterns, traffic distribution; no universal "error rate 0.95" for all order services. Per-user rate limiting must bound counter cardinality and TTL to prevent attacker exhausting memory with many distinct keys.

9. Discovery & Connection: Reuse Channel, Verify Actual Load Balancing

Kubernetes cross-namespace can use Service DNS; don't force Nacos just for cross-namespace. Registry center depends on cross-cluster discovery, existing governance, config platform needs.

Using Nacos: clarify namespace, group, service name, reachable registration address, health check, auth, credential rotation. Config subscribe failure ≠ discovery failure; separately observe registration status, discovery snapshot age, dynamic config version, subscribe errors.

Native grpc.NewClient("discovery:///inventory-service", ...) doesn't magically get Kratos service discovery: must register resolver or use Kratos Dial API with explicit registry.Discovery injection. Generally reuse long-lived ClientConn per target type, closed by assembly layer; no per-request connections, no returning pointers to generated client interfaces.

Keepalive for liveness probes, coordinate with server policy. Overly frequent PING may trigger GOAWAY; PermitWithoutStream and MaxConnectionAge not default must-haves. New pod no traffic: check endpoint updates, L4/L7 LB, HTTP/2 long-conn pinned to backend, resolver & balancer — not just tweak keepalive.

10. Observability: Explain One Order, Also Explain One Backlog

Trace records call graph and phase latency; logs persist state transitions, error causes, recovery actions; metrics serve trends, capacity, alerts. OTel Resource should have service name, version, deploy env; don't assume resource attrs auto-become all Prometheus labels.

Metric labels use low-cardinality dims: service, normalized operation, result category, controlled dependency name. Order ID, user ID, raw URL, dynamic peer address belong in logs or Trace, not universally on latency histograms.

API : Arrivals, completions, successes, rejections, P95/P99, attempt counts → Answers: Slow in processing or admission?

MySQL : Active conns, acquire wait, tx latency, lock wait → Answers: Does scale-out crush DB?

Outbox : Pending count, oldest age, lease expiry, failed terminal → Answers: Are accepted orders still progressing?

Inventory : Reserve latency, duplicate hits, release failures → Answers: Any resource leaks?

State Machine : PENDING_STOCK age, illegal transitions → Answers: Which orders cannot converge?

Runtime : Goroutines, heap, GC, CPU throttling → Answers: Queueing or resource constrained?

Outbox oldest age often closer to actual user wait than total backlog. One retry ≠ one new order. Async events propagate only necessary trace context; consumer may create consumption Span, long-delay replay/batch can use Span Link for correlation; forbid copying full headers, tokens, sensitive biz fields.

11. Config Hot Reload: Publish Immutable Snapshots, Show Effective Version

package runtimecfg

import (
  "errors"
  "sync/atomic"
)

type Snapshot struct {
  Version      string
  MaxInflight  int64
}

type Store struct{ value atomic.Pointer[Snapshot] }

func New(initial Snapshot) (*Store, error) {
  s := &Store{}
  if err := s.Update(initial); err != nil { return nil, err }
  return s, nil
}

func (s *Store) Update(next Snapshot) error {
  if next.Version == "" || next.MaxInflight < 1 { return errors.New("invalid runtime config") }
  copy := next
  s.value.Store(&copy)
  return nil
}

func (s *Store) Load() Snapshot { return *s.value.Load() }

Example only value fields, must be constructed via New; map, slice, pointer fields need deep copy, guarantee immutable after publish. Real update flow: parse full candidate config → validate field relations & ranges → build rules → atomic swap → log version & content digest; on failure continue running last verified snapshot.

Snapshot update ≠ immediate component effect. Lowering concurrency threshold cannot destroy semaphores held by in-flight requests; use adjustable-limit impl. Changing DB address should create new pool, drain old, not mutate a string. Watch reconnect semantics adapter-dependent; observe subscribe errors, last successful sync time, remote version.

12. Kubernetes: Stop Traffic, Drain, Recover Must Be Verified Together

12.1 Three Probes Each Own Responsibility

startupProbe

: initialization complete, avoid premature restart on slow start. livenessProbe: app in unrecoverable local fault; must not fail on shared DB brief blip. readinessProbe: instance should receive target business traffic now; draining phase must fail.

Readiness need not probe all deps every second. Shared dep failure may evict all pods simultaneously; design with degradability, health snapshots, hysteresis, failure policies.

replicas: 4
strategy:
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
template:
  spec:
    terminationGracePeriodSeconds: 60
    containers:
    - name: order
      image: registry.example.com/order:REPLACE_WITH_IMMUTABLE_TAG
      ports:
      - {name: http, containerPort: 8000}
      - {name: grpc, containerPort: 9000}
      - {name: admin, containerPort: 8081}
      resources:
        requests: {cpu: "2", memory: "1Gi"}
        limits: {cpu: "4", memory: "2Gi"}
      startupProbe:
        httpGet: {path: /startupz, port: admin}
        periodSeconds: 2
        failureThreshold: 30
      livenessProbe:
        httpGet: {path: /livez, port: admin}
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3
      readinessProbe:
        httpGet: {path: /readyz, port: admin}
        periodSeconds: 2
        timeoutSeconds: 1
        failureThreshold: 2
      lifecycle:
        preStop:
          httpGet: {path: /drain, port: admin}

Admin port must not be exposed to public internet. /startupz, /livez, /readyz, /drain all need app implementation.

12.2 /drain Must Flip State Fast, Not Hog Entire Graceful Termination Budget

Kubernetes runs preStop then sends termination signal to container; hook time counts in terminationGracePeriodSeconds. So /drain should be idempotent and quickly do three things: set draining flag, make readiness fail, stop Worker claiming new tasks; it must not long-wait for all HTTP/gRPC requests and Kafka sends to finish.

Drain executed by app after receiving termination signal, using independent deadline-bound shutdown context. Verifiable shutdown sequence: preStop or termination handler sets draining, rejects new create requests, stops claiming new tasks.

Leave bounded propagation window for K8s routing and registry to converge; existing connections may still send requests.

Graceful stop HTTP/gRPC, drain accepted requests; wait for in-flight Outbox batches until deadline.

Unfinished Worker tasks rely on lease recovery; never fake SENT.

Close producer, RPC, DB, finally flush telemetry within separate deadline.

60s budget can be split by real experiment: 5s state propagation, 40s request+Worker drain, 5s resource close, 10s margin; not universal constant, must verify via rolling deploy and long-connection experiments. EndpointSlice termination propagation async; readiness change won't stop existing HTTP/2 or gRPC connections immediately, so app-layer draining and client retry still required.

12.3 Go Runtime Resource Params Must Check Actual Effective Values

Go 1.25 introduces container-aware GOMAXPROCS: on Linux defaults to cgroup CPU bandwidth limit, not K8s CPU request, updates when visible CPU or cgroup limit changes; explicit GOMAXPROCS, runtime.GOMAXPROCS, or GODEBUG affect behavior. At startup log toolchain, GOMAXPROCS, CPU affinity, visible quota, memory config; monitor throttling in real container.

CPU request affects scheduling & resource contention share; CPU limit maps to usable CPU time cap. GOMAXPROCS parallelism ≠ kernel promised sustained CPU. GOMEMLIMIT is runtime soft limit, not container RSS hard cap; must leave headroom for non-heap, thread stacks, network buffers, native libs.

13. 8000 QPS: Calculate Workload First, Then Test Sustained Capacity

13.1 Fix Traffic Model First

Assume target 8000 req/s, 90% query, 10% create:

Query ~7200 QPS, create ~800 QPS.

Query cache hit 95% → ~360 QPS back to storage.

Each create writes order + Outbox → at least ~800 local tx/s; DB also handles indexes, redo, reserve result writeback, scans, consumer inbox, compensation.

Cannot be phrased as "8000 inventory-confirmed orders/sec", nor substitute cache-hit queries for create throughput.

13.2 Pod, Pool, N-1 Calculated Together

If single pod measured safe throughput at SLO = q, planned utilization = u, initial capacity lower bound: Pod count >= ceil(target QPS / (q * u)) Assumes balanced load and shared deps not saturated. N-1 fault tolerance requires remaining capacity after losing one pod still meets target. Max 20 pods × 50 DB conns = 1000 app conns theoretical, excluding Worker, migration, admin conns; horizontal scale likely hits DB first.

Little's Law estimates avg in-flight: 800 req/s × 50ms avg = ~40 avg in-flight. Cannot plug P99 into Little's Law for "precise pool size" because requests hold connections only in certain phases; must separately measure connection hold time and wait.

13.3 Read Optimization Must Not Break Read-Your-Writes

Query immediately after order create must read primary or path with explicit read-your-writes guarantee. Cannot have "create success but query misses" due to replica lag, let alone create another order. Staleness window for historical queries must be in contract; payment and inventory state not decided by local cache.

Kafka consumers sharing one group: messages partitioned among members, not broadcast to every pod. If events used to invalidate local cache, need independent consumption topology, other broadcast mechanism, or short TTL + version validation + compensation sync; even broadcast must handle disconnect and loss.

14. Load Test & Fault Acceptance: Make "Reliable" a Decidable Result

14.1 Four Baseline Load Tests

Pure create : Multi-user, ample inventory, new idempotency key each → Focus: Real persistence throughput.

Hot inventory : Single SKU high concurrency → Focus: Row locks, hotspot serialization, correctness.

Mixed query : Defined read/write ratio, cold/hot data → Focus: Cache benefit vs backsource cost.

Retry storm : Same key replay, disconnect, timeout → Focus: Idempotency & recovery cost.

Fixed single idempotency key mainly tests dedup query; all requests hitting same hot cached order mainly tests cache path. Using open arrival rate model: must record target arrival, actual sent, completed, biz success, rejected, timeout, dropped iterations; closed-loop model okay but must explain bias where rising latency reduces send rate.

14.2 Mandatory Context in Every Report

Report must include commit, image digest, Go & dep versions, pod count & resources, node specs, DB spec & pool, table data volume & indexes, Kafka partitions & ack policy, cache hit rate, request size & identity distribution. Warmup → ramp → stable observe → N-1 & rolling deploy; total duration covers connection rotation, GC, queue recovery, DB checkpoints.

Traffic model & total duration : Arrival model, read/write ratio, duration.

QPS : Arrival / completed / biz success / active reject.

Latency : P50 / P95 / P99 / timeout ratio.

DB : Connection wait, lock wait, redo/log pressure.

Outbox : Oldest age, stable period, recovery period, FAILED count.

Invariants : Duplicate orders, duplicate reservations, illegal transitions count.

N-1 : Capacity & SLO after single instance loss.

14.3 Eight Must-Automate Fault Experiments

Same-key concurrency : 100 concurrent CreateOrder → Pass: Only 1 order, 1 create event; different params → 409.

Crash after order commit : Kill process after commit success → Pass: Restart → order queryable; event eventually delivered.

Crash after Kafka ack : Kill Worker before SENT update → Pass: Allow duplicate msg; only 1 inventory effect.

Crash after consumer tx : Restart consumer before offset commit → Pass: Reservation & result event not double-produced.

Cancel arrives first : Send cancel then late reserve → Pass: No inventory decrement; order eventually cancelled.

Kafka blocked : Pause broker/network → Pass: At backlog threshold admit per policy; recovery → oldest age monotonically decreases.

DB slow : Inject latency / exhaust connections → Pass: Total conns within budget, admission guard works, liveness no storm.

Peak rolling update : Roll pods during create/consume peak → Pass: Committed requests recoverable, in-flight messages replayable, state eventually converges.

CI at minimum covers generated code consistency, compile, domain unit tests, real MySQL concurrent tx tests, message chain integration tests. go test -race ./... helps find data races on executed paths but cannot replace DB concurrency validation, nor prove all paths race-free.

15. Evolution Order: Each Step Eliminates One Explicit Risk

Phase 1. Consolidate monolith : Priority delivery: Contract, identity, idempotency key, transaction, reservation_id. Gate: Creation correctness tests pass.

Phase 2. Harden events : Priority delivery: Outbox, inbox, failure terminal, compensation, state machine. Gate: Crash, replay, out-of-order converge.

Phase 3. Deployment governance : Priority delivery: Deadline, admission, connection budget, probes, drain. Gate: Slow deps, releases controllable.

Phase 4. Performance pinpoint : Priority delivery: Trace, SQL, lock wait, cache, load test. Gate: Confirm & verify real bottlenecks.

Phase 5. Split on demand : Priority delivery: Inventory independent, cross-cluster discovery, Mesh. Gate: Independent scaling benefit > consistency cost.

When framework and Service Mesh coexist, must define single retry responsible layer, timeout propagation rules, sampling strategy. Before adding Dapr, Istio, more governance components, first answer: which existing ops cost does it solve, and who handles new failure modes?

Conclusion: Write Correctness as Constraints, Performance as Evidence

Kratos provides tools for organizing contracts, transport adapters, app lifecycle; order service reliability depends on whether those tools are realized as DB unique keys, local transactions, stable reservation_id, Outbox/Inbox, conditional state updates, failure terminal states, drain flows, and fault drills.

When same idempotency key retrieves same order, Kafka redelivery doesn't double-reserve, cancel-before-reserve still doesn't consume inventory, payment vs close-order race has defined convergence policy, Outbox blockage detectable and auditable recoverable, deploy & scale maintain connection budget and state convergence — only then system has foundation for high traffic.

As for 8000 QPS, answer must come from reproducible workload, latency distribution, biz success rate, and above fault experiments.

References & Verification Entry Points

Kratos v2 gRPC API: verify Middleware, UnaryInterceptor, StreamInterceptor, Discovery injection.

Kratos Recovery & Logging: verify middleware wrapping & error translation.

Google Wire: verify archived status & generation tool notes.

Go 1.25 Release Notes: verify container-aware GOMAXPROCS CPU limit, dynamic update, compat switches.

gRPC Keepalive: verify both-end PING policy & GOAWAY behavior.

Kubernetes Pod Lifecycle & Container Lifecycle Hooks: verify termination propagation & preStop time budget.