API Timeout After Ticket Creation: Why Query-First Recovery Beats Blind Retries

The Core Problem: Lost Success Responses

Increasing the create-ticket timeout from 5 seconds to 30 seconds may reduce slow-response timeouts, but it does not solve the fundamental issue: the success response can be lost in the network. The server has already created ticket W731, yet the caller only knows it received no response. If the tool returns a generic "creation failed, please retry", the model may invoke the create tool again, producing a duplicate ticket.

The principle: after a write timeout, preserve "result unknown" and let the Runtime decide based on the business contract. The model must not re-initiate a business action on an error message alone. The Runtime owns tool execution and recovery; business interfaces must provide verifiable results and deduplication boundaries. The real design question is not how many retries to allow, but what justifies sending a second write request.

Scenario: 5-Second Timeout, Ticket May Already Exist

A user asks to create a fault ticket for tenant A's device E17. The server generates W731, but the success response is lost. After 5 seconds the Runtime times out. From the business side the ticket exists; from the caller side the result is unknown. The caller must distinguish three judgments:

Reliable evidence the request was never executed.

Authoritative result proving it was executed.

Insufficient evidence — result unknown (the typical case after a disconnect).

Only when the request never left the local process and no other send attempt exists can we deem it unexecuted. Once handed to a proxy, SDK, or gateway, the request may still reach the business service. Therefore the result fed to the model should be " op-204 verifying, do not recreate", not a generic failure. The actual interception must happen in the execution layer.

Give Business Intent a Cross-Round Identity

Before the first send, the Runtime persists a business operation ID ( op-204) representing the user's approved create intent, bound to tenant, device, action, and a frozen parameter summary. This ID survives recovery, process switches, and conversation rounds. At least four distinct identities are involved:

Tool call ID — links a single model invocation to its return (OpenAI function_call_output uses call_id for protocol pairing, not persistent deduplication).

Business operation ID — identifies the business intent ( op-204).

Idempotency key — lets the interface recognize duplicate submissions ( k-204).

Attempt ID — distinguishes each actual network send for troubleshooting.

If the model generates a new call ID ( c2) on the next round, the execution layer must recognize it as a recovery of op-204 and return the operation status, not request a new creation. This identification relies on an explicit operation reference in the task, not on guessing from similar ticket titles. When the user genuinely wants another ticket, a new business operation ID must be issued to avoid over-deduplication swallowing legitimate requests. The Runtime must also store frozen request parameters; letting the model re-infer parameters (e.g., priority changing from P2 to P1) constitutes a parameter modification requiring separate authorization.

Idempotency Contracts Must Define Scope, Time, and Concurrency

Idempotency means duplicate submission of the same operation does not produce additional business side effects. An Idempotency-Key header is merely the entry point; how the interface identifies, retains, and handles duplicates determines recovery safety.

Stripe stores the first execution's status code and response body (including 500), retains keys for at least 24 hours, then cleans them; after cleanup the same key may be treated as a new request. Same key with changed parameters returns an error. However, parameter validation failures or conflicts with in-flight requests do not save an idempotent result because the endpoint hasn't started execution — "first request" cannot be universally defined as the first request to hit the server.

AWS EC2 RunInstances uses region-level or availability-zone-level idempotency depending on whether Placement.AvailabilityZone or SubnetId is specified. The same ClientToken across different boundaries can create distinct resources; failing over to another region does not automatically extend the original deduplication guarantee.

For the ticket interface, the uniqueness constraint should be "same tenant, same create action, same business operation ID". The replay key k-204 is only a transport identifier; the server must also verify a normalized parameter summary, not accept different content just because the key matches.

POST /tickets
Idempotency-Key: k-204
{"operation_id":"op-204","device_id":"E17","priority":"P2"}

GET /operations/op-204
{"status":"SUCCEEDED","resource_id":"W731","state_version":4,"replay_allowed":false}

Tenant identity comes from a trusted context, not from the model. The query returns business status, ticket ID, state version, and allowed next steps; replay permission is also server-verified, not a client-supplied boolean. Retention is split: request replay protection ≥ 24 hours, operation query records ≥ 7 days. Expiry of the former does not permit re-creation; as long as op-204 has a success record, W731 is returned. If deduplication evidence is lost, auto-write is refused and the case goes to verification. If operation records are also cleaned, the system must retain a basis to reject old requests (e.g., server-signed execution credentials binding tenant, action, operation ID, and deadline, or long-lived used-operation markers). The deadline covers the allowed recovery window; expired operations can only be verified, not treated as first-time creates. Concurrency: two executors submitting op-204 simultaneously — only one gets execution eligibility, the other receives "processing" or the existing result. Same key with different parameters is explicitly rejected. The article requires the business side to prove this guarantee; cross-system "exactly once" cannot be achieved with a single idempotency key.

Put Query-First into the Runtime State Machine

Separate two state categories:

Business result states: UNKNOWN, PROCESSING, SUCCEEDED, or explicit rejection evidence.

Scheduling states: pending send, retryable ( RETRY_READY), or manual review ( MANUAL_REVIEW). These do not change the business result; even after manual review, receiving success evidence does not auto-restore write permission.

The Runtime persists a PREPARED record before send, then conditionally updates to IN_FLIGHT with the attempt ID before handing to the network layer. Registration and actual send cannot be atomic; on recovery, any record that cannot prove it was never sent is treated as UNKNOWN. Even a local PREPARED state requires verifying the send path strictly followed register-then-send; a process crash must verify along the original operation ID, not restart due to missing success records.

For op-204, UNKNOWN triggers a query by operation ID:

Query returns SUCCEEDED + W731 → save result, return success to model.

Query returns PROCESSING → wait per server-suggested interval.

Query returns explicit business rejection → save failure reason, stop replay.

Query returns 404 → keep UNKNOWN (could be read-replica lag, wrong tenant, record cleanup, or request still in flight). The query interface must declare which authoritative state it reads; a generic ticket-list search "not found" is not proof of non-execution. RETRY_READY is entered via two paths:

Interface confirms NOT_EXECUTED and provides valid safe-replay conditions.

Result still unknown, but the contract guarantees same-operation replay is safe within current scope and retention.

Both paths must re-check current permissions and remaining budget. Query-first is not absolute: if the API already promises safe same-key replay with original result within scope/retention, the Runtime can replay directly. When an operation query exists, querying first avoids mistaking PROCESSING for failure and obtains the terminal business state.

Recovery needs a budget that survives process restarts. Example configuration: auto-recovery window 60 seconds, max 3 write attempts (including the first), query attempts separately limited with backoff and jitter. Expiry while still unknown →

MANUAL_REVIEW</sub>; do not mark "attempts exhausted" as business failure. Do not start a request if remaining time is insufficient; if <code>Retry-After

exceeds the remaining window, hand off. SDK, HTTP client, and Runtime retries must be unified into a single budget, with the Runtime as the sole scheduler to prevent double-counting. Multi-worker recovery of the same record uses version comparison or lease competition for execution rights; a lease is temporary scheduling ownership, not a substitute for business idempotency — a paused old process may still wake, and final duplicate prevention remains the business interface's responsibility.

Route by Execution Evidence, Not by Status Code

The most useful output of error classification is the allowed next action. Returning retryable=true to the model discards evidence and hands execution control back to text reasoning. Instead, the tool adapter should return a structured state, and the Runtime applies a fixed strategy:

Local validation failed, never sent → proves this attempt unexecuted → correct input, re-authorize if needed.

401/403 or business validation rejection → proves that response's rejection; old attempts verified separately → stop identical retry, handle auth/params.

Timeout, disconnect, gateway 502/504 → usually only know result unknown → query; limited replay only if idempotency contract satisfied.

429/503 → execution status depends on interface contract → confirm safety, then back off per budget.

202 or PROCESSING → accepted or still processing → query progress, do not create new business operation.

Same key different params, concurrency conflict → param mismatch or competing request → different params stop; conflict queries original operation.

Confirmed SUCCEEDED → business complete → return original ticket, no further writes.

The most dangerous misclassification is treating a gateway 504 as "transient failure, retry". The gateway didn't receive an upstream response; it did not undo the ticket. The correct action is to query business results via op-204, not bypass with a new key. Similarly, treating 500 as "retry will eventually succeed" is wrong — Stripe returns the cached 500 for the same key, meaning replay only re-fetches the first result. If the business result remains unclear, fall back to query or verification; changing the key might yield a different response but could also create another side effect. Auth errors must not trigger silent re-authentication; refreshing an expired credential is a controlled recovery action, but before resubmitting, current permissions must be re-verified. If an unknown attempt already exists, the new credential should first recover that operation, not start a new creation.

Manual Takeover Must Revoke Auto-Write Permission

Manual review is not just sending an alert. The Runtime marks op-204 as MANUAL_REVIEW, stops issuing new auto-send eligibility, and designates a reviewer. The handoff package includes: business operation ID, parameter summary, send history, obtained results, budget consumption, and current authorization state.

Query permission and write permission must be separate. The task may have been authorized to create tickets, but on recovery that authorization could be revoked. If query permission remains, the reviewer can verify W731 but cannot write again based on the old approval. Queries must also enforce tenant/resource permissions; knowing the operation ID must not become an unauthorized access path. The reviewer claims via conditional update on state version; the auto-send entry also checks the same state and execution generation (an incrementing version identifying the current executor). A conditional update on claim cannot revoke already-granted but not-yet-sent requests — these attempts must still be included in in-flight verification. To prevent even those from executing after takeover, a revocable execution token at the business execution boundary is needed, coordinated with eligibility judgment. Checking only at the send entry cannot stop requests already released; executed side effects cannot be rolled back by a revocation token.

Manual reviewers must not treat MANUAL_REVIEW as evidence the old request was cancelled; server-side same-operation deduplication remains essential. If a late response returns W731 after takeover, the system logs it as new evidence and notifies the reviewer to close, not letting the auto branch regain execution rights. If personnel see "timeout" and create another ticket, no software state machine can prevent human duplication. The human can finally confirm completion, confirm non-execution and resume per approved plan, or keep the case pending verification. "Result unknown" is a deliverable state; there is no need to force a success or failure just to make the task list all green.

Validate the Chain with Lost Responses and Expired Replays

Integration drills must deliberately separate "business success" from "call success". The following test designs target the contract; validation examines state records and actual ticket counts, not just HTTP logs.

Drop response after server creates W731 but before success returns. Expect Runtime enters UNKNOWN, queries by op-204, finds W731, transitions to success; tenant A's operation yields exactly one ticket; model gets no second create entry.

Inject 404 on first query, then restore. Expect UNKNOWN retained during 404; if a new operation ID is generated at that point, test fails immediately.

Block same create flow in PROCESSING . Verify query backoff and 60-second manual handoff, not looping creation; completed W731 must not revert to PROCESSING.

Expire k-204 replay protection, keep op-204 success record, resume old task. Expect return W731 or reject replay, not create new.

Simulate operation record also cleaned. Expect stop auto-write, require manual verification; send old request directly to business entry, verify expired execution credential or used-operation marker still rejects it.

Tests must cover SDK auto-replay paths, not just Runtime surface buttons. Especially check whether recovery code auto-generates k-205 because "old key expired" — for the same unknown business intent, changing the key is not a fix but a bypass of existing protection. After reset, concurrently send two same-operation same-param requests: verify one executes, the other queries or reuses result. Change second request's device to E18 → explicit rejection. Revoke write permission during UNKNOWN → system can still query within approved scope but not create. Additional competition drills: crash after persistent attempt registration but before send; old process pauses until lease expires, new process resumes, then old process wakes; auto-recovery preparing send while manual simultaneously claims. All must verify along original operation, producing no extra tickets. Post-takeover, no new auto-send eligibility; registered in-flight attempts and late results only add evidence, not restore auto execution. Preserve operation ID, attempt ID, execution generation, state transitions, and business record associations. Reporting success without authoritative evidence, or acquiring new send eligibility after takeover, are test failures.

Migrate from One Write Tool, Not a Universal Retryer

When moving to a custom system, start with the create_ticket tool: freeze inputs, add persistent operation records, make the adapter output "success, explicit not executed, unknown, processing", then run the above fault injections. This change delivers more value than adding three retries to every tool. If a third-party interface only offers create, with no stable operation identity or result query, shrink the automation scope: pause writes on unknown, go to manual reconciliation. External reference fields can be evaluated for lookup, but fuzzy search or eventually consistent lists cannot be directly upgraded to replay permission. The architect delivers a recoverable write operation contract, not just a model-callable function. The model expresses intent; the Runtime advances within evidence and budget; the business side provides results and deduplication guarantees; humans handle residual risk from insufficient evidence. This division applies to write operations where side effects must be traceable and duplication cost is high. If the interface cannot deliver identity, query, and deduplication boundaries, better to let a task stop at "pending verification" than to let one timeout become two tickets.