How OpenAI’s Responses API WebSocket Revamp Accelerates Agent Workflows by 40%

Background and bottleneck

OpenAI measured that an Agent workflow—model decides next action → client runs a tool → result returned to API → model continues—became limited by cumulative API request overhead once inference speed increased (GPT‑5 ~65 tokens/s, GPT‑5.3‑Codex‑Spark >1000 tokens/s). The measured composition of the bottleneck was:

API server overhead – request validation and dialogue‑state handling

Model inference – GPU token generation

Client overhead – tool execution and context building

Even with fast inference, each request rebuilt the full conversation context, repeated security checks, and re‑routed, adding noticeable latency.

Failed single‑request optimization

The first attempt optimized a single request by caching tokens and configuration in memory, skipping repeated tokenisation, reducing network hops, and speeding up the safety classifier. This cut first‑token response time (TTFT) by about 45 %, but every request still reconstructed the entire context, so the structural cost remained.

WebSocket persistent‑connection design

The solution removes the “rebuild context for every request” pattern by keeping a persistent connection that caches dialogue state on the server; subsequent requests only send the incremental input.

Two design options

Option A – treat the whole Agent rollout as a single long response. Using asyncio, the Responses API blocks in the sampling loop after a tool call, emits a response.done event, waits for the client to execute the tool, receives a response.append event, and then resumes sampling. This mirrors remote‑tool‑call behaviour but requires a massive API shape change and a full rewrite of existing client integrations.

Option B (chosen) – keep the existing API shape and add a previous_response_id field. Developers continue to call response.create with the same payload, adding only the generated previous_response_id. After the WebSocket connection is established, the server caches the conversation state identified by this ID, allowing the next request to reuse it.

The core difference: Option A changes the interaction model; Option B preserves HTTP semantics while swapping the transport layer to WebSocket.

What the state cache holds

When a request includes previous_response_id, the server retrieves from cache:

The previous response object

Earlier input and output items

Tool definitions and namespaces

Rendered tokens (avoiding duplicate tokenisation)

Successful model routing logic

Consequently, a new request only needs to send the “new input” rather than the full dialogue history, and security/classification checks run only on the incremental content. The cache lifetime is tied to the WebSocket connection; once the connection closes, the state is cleared.

Overlapping request execution

In HTTPS mode each request is independent and must complete all stages (validation → pre‑inference → sampling → post‑inference) before the next request can start, because the server has no prior context. With WebSocket, the cached state allows new requests to begin validation while previous post‑inference work (e.g., accounting, logging) is still running. Security classifiers and validators therefore process only the new input, removing a serial bottleneck.

The diagram (shown below) illustrates that the traditional pipeline forces a strict sequential order, whereas the WebSocket pipeline lets the stages of multiple requests overlap, effectively pipelining the work.

Why WebSocket instead of gRPC

The team evaluated gRPC bidirectional streaming but chose WebSocket primarily for developer experience. WebSocket is a simple message‑transport protocol that does not require changing the Responses API’s input/output schema; adding previous_response_id to response.create incurs minimal migration cost. gRPC would need proto definitions, client‑code generation, and substantial request‑logic rewrites, which the team deemed unnecessary for a speed‑up rather than a full architectural shift.

Production impact

After rolling out WebSocket mode, external feedback and metrics confirmed the benefits:

Codex : most Responses API traffic migrated; GPT‑5.3‑Codex‑Spark reached 1000 TPS with peaks over 4000 TPS; later models (GPT‑5.4, etc.) also benefited.

Vercel AI SDK : latency reduced by 40 %.

Cline : multi‑file workflow speed‑up of 39 %.

Cursor : OpenAI model speed‑up up to 30 %.

These numbers demonstrate that once inference became fast, API‑layer transmission efficiency turned into the primary bottleneck, and the persistent WebSocket connection resolved it.

Takeaways

Optimization focus must follow the dominant bottleneck: first improve inference, then address API overhead when inference speeds up. Maintaining API compatibility proved worthwhile; although Option A was theoretically superior, Option B offered a pragmatic balance of migration cost and performance gain. Cache invalidation must be considered—state is lost on connection loss or timeout, so long‑running Agent tasks need recovery mechanisms. Transport‑layer improvements are essential to unlock the full value of faster models.

Official API documentation: https://developers.openai.com/api/docs/guides/websocket-mode