Why a Single Timeout Spawned Two Risk Reviews: Production MCP Server Patterns

When LLMs gain write permissions to enterprise systems, the real problem isn't "how to register a Tool", but how to turn an untrusted, replayable, potentially timed-out tool call into a trusted and recoverable business operation.

1. Incident: A Seemingly Ordinary Retry

An e-commerce support assistant receives a request to trigger a risk review for order O20260728001 via the request_risk_review tool. The timeline:

10:31:08.120  Agent initiates tools/call(request_risk_review)
10:31:08.241  MCP Server creates local request R10086, calls Risk API
10:31:10.742  Risk API response times out
10:31:10.743  MCP Server returns "submission failed"
10:31:12.000  Agent automatically retries same intent
10:31:12.531  Risk API creates another Review

The problem: the first call may have succeeded but the response was lost. Mapping timeout directly to FAILED turns uncertainty into duplicate side-effects. A production-grade MCP Server is not a thin proxy forwarding tools/call to an internal HTTP API; it converts model-generated Tool Intent into constrained business commands.

LLM / Agent
       │
Untrusted Tool Intent
       │
       ▼
┌──────────── MCP Protocol Boundary ────────────┐
│ tools/call · schema · transport · auth         │
└─────────────────────┬─────────────────────────┘
                      │
                      ▼
┌──────────── Business Safety Boundary ─────────┐
│ Subject · Authorization · Validation           │
│ Idempotency · State Machine · Recovery · Audit │
└─────────────────────┬─────────────────────────┘
                      │
                      ▼
              Order / Risk System

MCP handles capability interoperability; it does not vouch for order authorization, business uniqueness, transaction recovery, or eventual consistency. The latter must be implemented by the business safety boundary.

2. MCP 2026: Protocol Stateless ≠ Business Stateless

MCP 2026-07-28 still uses JSON-RPC 2.0 but the protocol core is now stateless: initialize / initialized and Mcp-Session-Id are removed. Each request carries protocol version, client info, and capabilities; clients may optionally use server/discover to pre-discover capabilities, but it's not a prerequisite for every call. A simple round-robin load balancer can route requests to any replica without sticky routing or shared session storage.

This does not mean business cannot persist state across calls. The correct approach is to have tools explicitly return business handles (e.g., review_id, cart_id, workflow_id) that subsequent tools accept as parameters:

tools/call(request_risk_review)
                │
                └── { review_id: "R10086", status: "UNKNOWN" }

tools/call(get_risk_review_status, { review_id: "R10086" })

State then belongs to the business object — auditable, durable, recoverable across replicas — rather than hidden in a TCP connection or MCP session. For interactions requiring user input/confirmation, the 2026 spec uses Multi Round-Trip Requests (MRTR), avoiding long-lived bidirectional streams. The old HTTP+SSE transport is deprecated; new implementations should baseline on Streamable HTTP.

3. Schema Is the First Constraint, Not the Security Boundary

A tool's inputSchema is a shared contract between model, host, and server. The 2026 spec defaults to JSON Schema 2020-12 and supports outputSchema and structuredContent. additionalProperties: false lets models and clients catch invalid fields early, but it is not an authorization system: a malicious client or model can still craft a valid yet unauthorized order_id.

{
  "name": "request_risk_review",
  "description": "Request a risk review for an order the caller may operate.",
  "inputSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "order_id": {"type": "string", "minLength": 1, "maxLength": 64},
      "reason": {"type": "string", "minLength": 1, "maxLength": 500}
    },
    "required": ["order_id", "reason"]
  },
  "outputSchema": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "review_id": {"type": "string"},
      "status": {"enum": ["PENDING", "SUCCEEDED", "FAILED", "UNKNOWN"]}
    },
    "required": ["review_id", "status"]
  }
}

Note the schema lacks user_id, operator_role, approved, or any downstream topic — these must be decided by the server. The full chain:

MCP Schema
  ↓  structure, types, length
Go DTO
  ↓  format, domain rules, object existence
Domain Validation
  ↓  can current subject operate this order
Authorization

func requestRiskReview(ctx context.Context, args ReviewArgs) (mcp.CallToolResult, error) {
    subject, ok := SubjectFromContext(ctx) // verified token / session export
    if !ok {
        return businessError("UNAUTHENTICATED", "Please authenticate first"), nil
    }
    if !validOrderID(args.OrderID) || utf8.RuneCountInString(args.Reason) > 500 {
        return businessError("INVALID_ARGUMENT", "Invalid order ID or reason format"), nil
    }
    if !policy.CanRequestRiskReview(subject, args.OrderID) {
        return businessError("FORBIDDEN", "No permission to request review for this order"), nil
    }
    return submitReview(ctx, subject, args)
}

Caller identity (which MCP client) and end-user identity (who the business operation represents) must be separated: the former for ingress auth, quotas, audit; the latter for order-level authorization. Both must come from trusted context, not model parameters.

4. Distinguishing Protocol Errors from Business Execution Errors

This boundary directly affects host behavior.

Issue                                    | Layer              | Correct Expression
-----------------------------------------|--------------------|------------------------------
JSON-RPC format error, unknown method    | Protocol           | JSON-RPC error
Tool name not exist or unauthorized      | Protocol/Discovery | JSON-RPC error or reject
Invalid param, order not exist, no perm  | Business Execution | Tool Result, isError: true
Downstream unavailable, UNKNOWN state    | Business Execution | Tool Result, explicit queryable status/error code

Thus businessError() must not covertly construct a JSON-RPC error; it should return a valid Tool Result with isError: true so the model understands whether to correct parameters or query status. Protocol parse failures must not masquerade as "risk submission failed". The Tools spec explicitly distinguishes JSON-RPC layer errors from tool execution results.

For HTTP servers, verify token audience, reject tokens not targeting this MCP Server; do not forward the client's token to downstream. The 2026 version further requires issuer validation in auth code flow and pushes Client ID Metadata Documents (CIMD) to replace deprecated DCR.

5. Request Idempotency ≠ Business Uniqueness

Using only an idempotency_key risks a second error: it only answers "is this the same request replayed", not "does business allow two valid reviews to coexist". Production design uses dual keys:

request_key  = unique identifier of this user click / Agent Command
business_key = order_id + review_type + active_generation

request_key

prevents duplicate execution of the same call; same key must return the first-created record or current status. business_key protects business invariant: at most one active review per order, review type, and active generation.

CREATE TABLE risk_review_request (
    id              BIGINT PRIMARY KEY,
    request_key     VARCHAR(128) NOT NULL,
    business_key    VARCHAR(128) NOT NULL,
    order_id        VARCHAR(64)  NOT NULL,
    requester_id    VARCHAR(64)  NOT NULL,
    status          VARCHAR(32)  NOT NULL,
    downstream_ref  VARCHAR(128),
    retry_count     INT NOT NULL DEFAULT 0,
    next_retry_at   TIMESTAMP NULL,
    version         BIGINT NOT NULL DEFAULT 0,
    created_at      TIMESTAMP NOT NULL,
    updated_at      TIMESTAMP NOT NULL,
    UNIQUE KEY uq_request_key (request_key),
    UNIQUE KEY uq_business_key (business_key)
);

Unique constraints are the final arbiter. They are more correct than "check-then-insert" (vulnerable to concurrent penetration) and closer to business facts than Redis distributed locks. Locks only temporarily reduce contention; you still handle leases, process pauses, failover, and accidental release. Unique constraints make the invariant part of the data model. Only when a cross-resource critical section truly cannot be solved by atomic state transitions or data constraints should a distributed lock be introduced.

Creation path: on any unique conflict, read existing record and return; never call Risk API again due to conflict:

func submitReview(ctx context.Context, subject Subject, args ReviewArgs) (mcp.CallToolResult, error) {
    keys := deriveKeys(subject, args, RequestKeyFromContext(ctx))
    req, inserted, err := repo.InsertIfAbsent(ctx, NewReviewRequest(keys, subject, args))
    if err != nil { return businessError("TEMPORARY_UNAVAILABLE", "Temporarily unable to create review request"), nil }
    if !inserted { return renderReview(req), nil }

    // Pass request_key to downstream; downstream must deduplicate or query by this key.
    result, err := riskClient.CreateReview(ctx, req.RequestKey, args)
    if err == nil {
        _ = repo.MarkSucceeded(ctx, req.ID, result.Reference)
        return renderReview(req.WithSuccess(result.Reference)), nil
    }
    return classifyCreateFailure(ctx, req, err)
}

6. UNKNOWN Is Incomplete Fact, Not Failure Alias

PENDING

, FAILED, UNKNOWN, and RETRYABLE are not interchangeable. The first two describe known outcomes; UNKNOWN means downstream may have executed but local proof is missing; RETRYABLE means a query proved downstream has not yet created, so a controlled resend is allowed.

Current State | Event                          | Next State | Allow Create Again?
--------------|--------------------------------|------------|------------------
PENDING       | Create succeeded               | SUCCEEDED  | No
PENDING       | Explicit business rejection    | FAILED     | No
PENDING       | timeout / reset / transient 5xx| UNKNOWN    | No
UNKNOWN       | FindByKey found                | SUCCEEDED  | No
UNKNOWN       | Downstream explicitly not exist| RETRYABLE  | Only Worker-controlled
UNKNOWN       | Query still times out          | UNKNOWN    | No
RETRYABLE     | Worker successfully claimed    | PENDING    | Only that Worker

Recovery hinges on legal state transitions, not "retry count < 3":

func reconcileUnknown(ctx context.Context, req ReviewRequest) error {
    remote, err := riskClient.FindByRequestKey(ctx, req.RequestKey)
    switch {
    case err == nil && remote.Found:
        return repo.MarkSucceeded(ctx, req.ID, remote.Reference)
    case err == nil && !remote.Found:
        return repo.MarkRetryable(ctx, req.ID, nextBackoff(req.RetryCount))
    default:
        return repo.RecordProbeFailure(ctx, req.ID, err) // keep UNKNOWN
    }
}

If Risk API lacks query-by-business-key or idempotent create, the MCP Server cannot guarantee no duplicate creates. Then modify downstream contract, add reconcilable query interface, or turn high-risk writes into explicit human confirmation; "catch timeout then resend" is not acceptable.

7. Multi-Instance Workers: Claim via Conditional Update, Not Default Distributed Locks

Recovery workers scale horizontally, but each RETRYABLE record can be claimed by only one worker. At small scale, version-based CAS suffices:

UPDATE risk_review_request
SET status = 'PENDING',
    retry_count = retry_count + 1,
    version = version + 1,
    updated_at = NOW()
WHERE id = :id
  AND status = 'RETRYABLE'
  AND version = :version
  AND next_retry_at <= NOW();

Only the worker with affected_rows = 1 gains send rights. At higher volume, use FOR UPDATE SKIP LOCKED in a short transaction to batch-claim; commit immediately, then call remote service outside the transaction to avoid holding DB locks during network waits.

Worker crash after send may leave UNKNOWN — this is not a bug but a failure path the state machine must cover. On restart, query downstream by request_key first, then decide whether to proceed. DB CAS solves "who has execution right"; downstream business key solves "will execution duplicate"; both are indispensable.

8. Long Tasks: MCP Tasks Don't Replace Business State Machine

For long-running risk reviews, regular clients can use explicit review_id + get_risk_review_status. Hosts supporting the io.modelcontextprotocol/tasks extension can have tools/call return an MCP Task, then use tasks/get, tasks/update, tasks/cancel to express protocol-level long-task lifecycle.

MCP Task                          risk_review_request
Protocol layer: how Host observes  Business layer: review executes once, completes
Cancellable, updatable, pollable   Idempotent, authorized, UNKNOWN, reconciliation, audit

Even with Tasks, dual keys, unique constraints, and UNKNOWN recovery remain necessary. Conversely, even if Host doesn't yet support Tasks, the business state machine already works correctly. This is the benefit of using extensions rather than stuffing business facts into protocol sessions.

9. High Concurrency: Budget, Backpressure, Downstream Protection

"20 pods × 64 goroutines" is not a capacity model. The downstream Risk API's safe concurrency is the hard boundary. If it's verified to handle 100 concurrent requests, 20×64=1,280 inbound concurrency only amplifies failure 12×.

First establish a discussable deadline budget (example numbers, not performance commitments):

Client deadline:          5.0s
├── Auth / Policy:        0.1s
├── DB:                   0.3s
├── Risk API:             2.5s
└── reserve:              2.1s (queueing, serialization, error return)

Then set three explicit gates for write tools:

Max concurrency : per-instance in-flight ceiling to prevent unbounded connections, memory, goroutines.

Queue capacity : max allowed queued requests; when full, return overload result fast instead of silent backlog.

Global downstream capacity : cross-replica total quota coordinated by actual downstream capacity; implemented via gateway rate-limiting, shared quota service, or downstream's own limits — choice depends on whether global protection is truly needed.

Little's Law gives intuition: in-flight ≈ arrival rate × residence time. When Risk API slows, residence time rises, in-flight count grows even if arrival rate is steady. Correct actions: timeout, limit concurrency, backpressure, fast-fail — not unlimited goroutines or blind replica scaling.

Caching, MQ, Outbox, Kubernetes are not prerequisites. Introduce task workers only when synchronous wait exceeds interaction budget; evaluate Outbox only when reliable propagation to multiple consumers becomes a real need and DB+MQ dual-write window is a real problem; assess caching only when reads are a proven bottleneck and stale data is acceptable.

10. Observability: Can You Reconstruct a Tool Call?

During disputes, engineers must answer: who called which tool? representing which end-user? why did policy allow? which request_key / business_key? did downstream execute? why stuck in UNKNOWN?

Every call must associate at least: request_id, sanitized request_key, business_key, review_id;

Client principal, end-user principal, tool name, policy decision;

Parameter summary or hash — not full order content, reason text, tokens;

State transitions, downstream latency, error classification, probe count, worker claim result.

Prioritize actionable alerts: UNKNOWN stalling too long, abnormal auth rejection rate, rising downstream latency, queue saturation, anomalous CAS claim failures. Collecting only QPS, CPU, and log volume cannot answer "was this review actually created?".

11. Verify Invariants with Fault Injection

Without real load-test data, don't claim QPS or P99; but write clear acceptance tests. Before launch, run at least these integration tests:

Injected Condition                                    | Expected Result
------------------------------------------------------|--------------------------------------------------
Two replicas submit same request_key                  | Only one request record; both return same business result
Different request_key, same business_key              | Only one active review; second reads existing object
Risk API created but response dropped                 | Local enters UNKNOWN; recovery converges to SUCCEEDED, no re-Create
FindByKey confirms not exist                          | Only one CAS-success worker can resend
Forge another user's order_id                         | Tool Result = business rejection; no downstream call
Queue and concurrency slots exhausted                 | Fast return understandable overload result; no unbounded pile-up

These tests verify not "SDK runs" but that under the most dangerous failures, invariants hold: a single business fact will not be incorrectly changed twice due to model retries, network packet loss, or multi-replica concurrency.

12. Evolve from Minimal Correct Version

Read-only V1 : Streamable HTTP, auth, order-level authorization, schema/domain validation, audit, deadline; no cache, MQ, or service mesh needed.

Write V2 : Dual keys, unique constraints, explicit review_id, downstream idempotent/query contract, UNKNOWN state machine.

Long-running V3 : Status query, recovery workers; layer MCP Tasks when Host supports, but don't migrate business state to protocol.

Reliable Events V4 : Only when reliable propagation to multiple consumers becomes an explicit requirement, evaluate Outbox/MQ and handle dual-write and recovery windows.

Conclusion

The maturity of a production-grade MCP Server is not determined by how many internal systems it connects or how much middleware it deploys, but by whether it can converge untrusted model intent into trusted business facts.

Get identity, authorization, dual-key invariants, and UNKNOWN recovery right first; then let workers, Tasks, rate-limiting, and scaling serve those facts. Stateless protocol simplifies deployment; business state machine ensures correctness. Each doing its job is the true foundation for enterprise MCP Tools to go live.

Core Conclusions

MCP is an interoperability protocol, not a framework for business idempotency, authorization, or transaction recovery.

2026-07-28 adopts a stateless protocol core; cross-call state should use explicit business handles, not protocol sessions.

Schema is the first input constraint; handler domain validation and server-side authorization are the security boundary. request_key handles call replay; business_key protects business uniqueness; the two must not be mixed.

Timeout is not failure. When downstream may have executed, enter UNKNOWN, query first, then decide whether to resend.

Multi-instance recovery uses DB CAS/row locks to claim tasks; don't default to Redis distributed locks.

MCP Tasks solve how long tasks express themselves to Host; they don't replace business state machine and reconciliation.

High concurrency core is downstream capacity, deadline, concurrency ceiling, and backpressure — not more goroutines or pods.