How OpenAI Cuts Agent Loop Costs: Harness, API & Inference Optimizations
This article details OpenAI's multi-layer efficiency techniques for agentic AI systems, covering harness-layer WebSocket persistence and prompt caching, API-layer incremental tokenization and parallel safety checks, and inference-layer cache-aware routing, KV cache management, speculative decoding, and prefill-decode separation to reduce per-task costs.
Agentic AI Application Structure
When you assign a task to Codex or ChatGPT Work (e.g., "trace the checkout regression, patch it, and run the tests"), the query does not go directly to the LLM. An AI application like Codex is a system of many components; the LLM is only one of them. Before the LLM sees a single token, the user query passes through several layers.
An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and outputs a sequence of tokens. It cannot run shell commands, edit files, or remember anything between calls. Agent tasks like "fix this bug and run tests" consist mainly of actions. Something must turn the model's predicted tokens into real commands, feed results back, and continue until the task is done.
That is the job of the harness layer : the system built on top of the LLM that handles these responsibilities. It takes the user task as input, decides which instructions, tool definitions, and how much history to include in context, and maintains conversation history. When the model returns a tool call, the harness executes it in a sandbox under an approval policy, appends the result, and sends the conversation back to the LLM.
Even the request from the harness does not go straight to the LLM. In production-grade applications, an API layer (application layer) sits between the harness and the inference endpoint. This layer exists because real products need to handle things the harness or LLM itself does not: authenticate callers, enforce rate limits, and — crucially — convert text to the token IDs the LLM expects and convert generated tokens back to text.
When the API layer has prepared the context and tokenized it into a stream of token IDs, it reaches the inference layer . This is a remote endpoint backed by large GPU clusters that host the model. Its job is to run the model computation on the prepared tokens as efficiently as possible and generate a response — whether that response is a new tool call or a final answer — then hand the generated tokens back to the API layer.
What Happens When You Send a Request to an AI Agent
Using the Codex example ("trace the checkout regression, patch it, and run the tests"), here is the flow:
The harness assembles instructions, tool definitions, and your task into a request and sends it to the API.
The API buffers the request in memory, parses JSON, and validates it: well-formed request, chosen model supports every requested capability.
It checks caller identity, applies rate limits, and runs pre-flight checks.
The API renders the conversation into the model's input format and tokenizes it.
The API dispatches tokens to the inference layer while simultaneously starting safety checks: classifiers look for cyberattacks, bioweapon content, etc. Safety checks race to finish before the first generated token returns.
The inference layer processes the prompt and begins generating. The output here is not a final answer; it is a tool call: search the codebase for "checkout timeout".
Generated tokens flow back to the API.
The API converts tokens to text and wraps them into API events.
The API streams these events to the harness.
The harness recognizes the tool call, runs the search in its sandbox, appends the output to the conversation, and sends the updated conversation back to the API.
This is just one iteration. In practice, a task repeats this loop many times until the model produces a final summary and the orchestration layer returns control to the user. Every iteration repeats much of the same work: history is resent, text is re-tokenized, the prompt is reprocessed. Eliminating this repeated work is the primary optimization opportunity.
Orchestration Layer Optimizations
The orchestration layer (harness) is the layer closest to the user; it turns the raw user request into context and loops with the LLM until the task is done. It has two main responsibilities: it is the source of truth for the conversation (authoritative record of every instruction, message, tool call, and tool result), and it runs the agent loop (decides what goes into context, sends requests, streams responses, parses tool calls, executes them in a sandbox under approval policy, appends results, and repeats). Heavy tasks can theoretically loop 100+ times; each iteration adds overhead — e.g., an extra 1 second per model call adds ~30 seconds to a long task.
From the harness perspective, latency comes from four sources: network, prompt processing, context content, and loop round-trips. OpenAI uses four techniques at this layer:
1. Persistent WebSocket and Incremental Requests
The harness runs on the user's machine, but the model runs in OpenAI's data centers, so every model call is a network exchange. The standard way is HTTPS. The harness opens a connection, sends a request containing everything the server (API layer) needs, then receives the response. Because LLM responses arrive incrementally (one token at a time), chat apps typically use Server-Sent Events (SSE) over HTTPS so the client sends one request and the server streams the answer back in small chunks. SSE is a one-way street, great for the chat era: one user message → one model call → one streamed answer. But agents are not one-shot. A single Codex turn may contain many model calls. With HTTPS, every call is a new request, incurring two independent costs:
Connection establishment: Opening a new HTTPS connection means a TCP handshake then a TLS handshake — multiple network round-trips before any useful payload bytes move. Paying this once per user message is acceptable; paying it repeatedly inside a single conversation turn is expensive.
Payload duplication: HTTP is stateless, so every request must carry everything the server needs. Instructions, tool definitions, and the entire conversation so far must be in the request. By the 20th tool call, the harness re-sends the original prompt, 19 tool calls, and 19 tool results just to append one new result. Payload grows with each iteration, so the harness spends more and more time uploading data the server has already seen.
Solution for connection cost: open one connection and keep it alive instead of creating a new one per call. That is exactly what WebSocket is designed for. WebSocket requires one initial handshake; afterward either side can send messages at any time with no additional establishment. Codex opens a WebSocket to the API and keeps it open for all model calls in a turn, eliminating repeated TCP/TLS setup.
Solution for payload duplication: stop re-sending what the server already knows. The harness retains the previous request and the completed response. If nothing has changed except the conversation input, it sends only the new entries plus a reference to the previous response. After a tool call, the next message on the socket can be as small as the new tool result. The server reconstructs full context from its saved state, so the model still sees everything; only the bytes on the wire change.
2. Stable Prompt Prefix
LLM providers use a technique called prompt caching to avoid recomputation. When a prompt arrives, the model reuses cached internal state (the prefix) for the leading portion that matches a previous request, computing only the remainder. Matching is exact, token-by-token. If you change one token early in the prompt, everything after it must be recomputed.
For the harness, this means the prompt it builds each call must start with exactly the same bytes as the previous call. Sounds simple, but the harness rebuilds the request from live memory state every time, so any tiny difference in assembly can silently break the match. OpenAI shared an example: Codex stored MCP tool definitions in a hash map, which does not guarantee order, so the same set of tools could serialize in a different order each request. The tools in context were identical, just reordered. Codex still completed the task, but at higher cost.
The efficiency trick: treat history as append-only and keep volatile runtime state (like approval settings) out of the prompt. For example, when a user changes approval settings mid-session, the harness does not edit tool definitions in the prompt; it applies the new policy at the next tool run. The prompt stays stable, so the cache stays valid.
3. Deferred Tool Discovery
Stable prompt prefixes make repeated context cheaper, but they don't keep the prompt small. In agents with access to many tools, this can be a problem. Tool schemas alone can consume huge space: once you count connected integrations and MCP servers, a Codex session can expose hundreds of tools. Each tool schema is a JSON object with name, description, and parameter definitions. Once all of that enters the prompt, the LLM must process it in computation, even though many tools may be irrelevant to the current task.
To avoid putting unused tools into the prompt, OpenAI uses deferred discovery . The prompt carries only core tools (e.g., shell and file editing) plus a tool_search tool. The other hundreds of integration definitions stay out of the prompt entirely. When the model needs a capability, it calls the search tool with a query like "list deployments". The harness searches the registry and loads matching definitions into context. Search uses BM25, a lexical ranking algorithm that finds tools whose descriptions overlap with the query terms.
Beyond deferred discovery, two more techniques keep context lean: schema compression strips descriptions and folds nested structures while preserving parameter names, trimming oversized tool schemas to a token budget; conversation compression condenses long history into a short summary.
4. Code Mode
In the default setup, the model emits one tool call, waits for the result, reasons, then emits the next call. In many scenarios, no reasoning is needed between those calls; the model just emits calls one by one to gather results. This is expensive because each tool call requires a full model round-trip. Moreover, after each tool call its result is added to context, wasting context space.
With Code Mode , the model writes a small program to perform those calls instead of emitting them one by one. The harness executes the program in an embedded JavaScript runtime where every tool is available as a function. The script can fan out independent calls in parallel, filter and merge results with ordinary code, and return only the distilled answer. Intermediate data stays in the runtime; only the final result enters context.
API Layer Optimizations
The API layer is the service the harness talks to. It sits between the harness and the inference layer, runs on commodity CPUs. The harness sends JSON describing structured conversation items; the model consumes and produces token IDs. When a request arrives, it goes through these steps:
Request body is buffered into memory.
JSON is parsed and validated against schema (fields well-formed, arrays in expected ranges, etc.).
Second validation pass checks semantics: does the chosen model support the requested features?
Pre-flight checks: auth and entitlements, rate limits, and any image checks in the request.
API renders and tokenizes the conversation, then kicks off two tasks in parallel: the inference request to GPUs, and safety checks on the prompt.
As generated tokens stream back from the inference layer, the API converts each token to text, wraps it into an API event, and streams it to the client.
These steps repeat on every loop iteration. The API cannot make GPUs faster; it can only add as little latency as possible before and after inference. Time the API spends parsing, validating, or tokenizing becomes extra latency the user feels.
1. Tokenize Only the Delta
Models don't read text directly. Before inference starts, the prompt must be converted to token IDs. The tokenizer runs in O(n) — linear pass over every token in the prompt, replacing each with its token ID. Under stateless HTTP, every call re-tokenizes the entire conversation. By the 20th tool call, the API is re-reading tens of thousands of tokens just to extract one new result. The GPU only needs the new tool result, but the CPU re-reads the whole book from page one, and cost grows with input length.
With WebSocket, the API keeps the already-tokenized conversation in server memory. The first request tokenizes the full prompt; every subsequent request sends only the new material. The API tokenizes just that delta and appends it to the stored sequence. Tokenization overhead per call no longer depends on conversation length; it becomes closer to O(1), depending only on the marginal input length.
2. Run Safety Checks in Parallel with Inference
Agents must safety-check every request before showing output to the user. OpenAI implements safety checks in the API layer. Images go through dedicated classifiers; prompt text is fed to classification models that flag dangerous content (e.g., cyberattacks, bioweapon instructions). These classifiers take time. A naive design runs them first, starts inference only after they pass. The problem: that latency adds to time-to-first-token (TTFT) for every request, even though the vast majority are benign.
The fix: run safety checks and inference simultaneously. The model needs some time to process the prompt before emitting the first token, so safety checks can use that window. If a check fails, the API reacts differently per model: for some models it streams tokens to the user immediately and cuts the stream on failure; for more sensitive models it buffers output until checks pass. Either way, the time spent on safety checks is hidden inside the wait that would happen anyway.
3. Route Traffic to Newer CPUs
When a company runs servers at scale, its fleet eventually contains machines purchased in different years, hence different CPU generations. Scheduling software usually treats same-labeled machines as identical. In reality they are not. OpenAI queried Kubernetes for the actual processor model behind each deployment and found the same machine label mixed older Broadwell chips with newer Ice Lake chips. The older chips were ~20% slower on time-to-first-token for the same traffic and consumed ~2x CPU utilization.
To fix this, OpenAI shifted more traffic to machines with newer processors and made CPU generation an explicit factor in capacity planning. Sometimes the best software optimization is better hardware.
Inference Layer Optimizations
The inference layer is where the model actually runs. It has GPU and accelerator fleets, plus the serving software (engine) that drives them. The model's core computation is massive matrix math that can be parallelized across thousands of cores. This layer schedules and batches incoming requests across the fleet, saves the per-conversation state that generation depends on, executes the model's forward pass, and hands each generated token back to the API layer.
Demand for tokens grows faster than hardware can scale, so at this layer slow and wasteful are the same thing. Waste hides in four places: work routed to the wrong machine; memory holding the wrong state; parallel hardware sitting idle during sequential generation; and two mismatched stages sharing the same machines.
1. Balance Load with Cache-Aware Routing
OpenAI serves models on large GPU fleets. Every request must be sent to one machine. If routing is unbalanced, some machines idle while others queue. Idle GPUs are the most expensive waste in the stack. A second, less obvious problem: each machine caches the computed state of conversations it recently served. If the next request for a conversation lands on a different machine, that cache is useless. The new machine must recompute from scratch, even though the work was already done elsewhere.
Therefore the router has two goals:
Spread load evenly: send requests to the least busy machine.
Use the cache: send requests back to the machine that already "knows" the conversation.
OpenAI's router weighs both on every request, along with base factors like geography and available capacity. Load-balancing improvements alone have significantly reduced model serving cost.
2. Manage the KV Cache
When a Transformer generates a token, it attends to all previous tokens. To avoid recomputing those attention states for every new token, the serving system saves them in accelerator memory — the KV cache . The cache grows linearly with conversation length, multiplied by every concurrent conversation the cluster serves. For long contexts it can approach the size of the weights themselves. If software evicts the wrong state, the inference layer must pay the full processing cost again when that conversation returns.
The efficiency technique: manage the cache based on real usage. OpenAI analyzes production traces to learn which cache entries are likely to be needed again, uses that data (not intuition) to test eviction policies, and optimizes how cache state is stored and moved between memory tiers. This ensures the most valuable computed state stays hot longer without exhausting memory or wasting time moving data.
3. Speculative Decoding
Generation is sequential: each token depends on the previous one, so even though a model can digest a huge prompt in one parallel pass, it must still generate the answer one token at a time. In large models, generating those tokens is expensive and slow, even when the next token is easy to guess. For an input like "the capital of France is", the next token is almost certainly "Paris", but the model still processes the entire conversation and runs massive matrix multiplies to predict it.
Speculative decoding uses a smaller draft model to propose the next several tokens, then the large model verifies all of them in one parallel pass. In many cases the draft's proposals are correct and accepted by the large model, saving the time the large model would have spent generating those tokens one by one. When a proposal is wrong, the large model discards it and generates its own token, so output quality does not change. The key metric for evaluating a draft model is mean acceptance length — how many proposed tokens the main model accepts per forward pass.
4. Separate Prefill and Decode
Inference has two stages: prefill and decode . Prefill processes the entire incoming prompt in one large parallel pass and builds the internal state (KV cache) the model needs. Decode then generates the response one token at a time. These are very different workloads. Prefill is compute-bound; decode is memory-bound, spending most of its time streaming weights and cache state through memory. Running both on the same machines means neither runs on hardware optimized for its bottleneck.
The efficiency move: separate them. A portion of the cluster handles prefill, another portion handles decode, each configured for its own bottleneck. When a request finishes prefill, its computed state (KV cache) is transferred to a decode machine, which then generates the response token by token.
Summary of Efficiency Techniques Across Layers
The harness avoids re-sending what the server already has; the API avoids re-processing what it already processed; inference avoids re-computing what it already computed.
Lessons from OpenAI's Push for Efficiency and Lower Cost per Successful Task
Zooming out, these cross-layer optimizations all try to avoid paying twice for the same work. The harness sends only deltas and keeps prefixes stable so caches stay valid. The API tokenizes only the delta and hides inevitable work inside windows that must happen anyway. Inference routes conversations back to their cached state instead of recomputing. The layers also reinforce each other: when the harness keeps caches intact and the API improves cache hit rates, the savings show up on GPUs, meaning lower cost for users. Beyond the techniques, OpenAI engineers shared these lessons:
Keep design simple. Favor simplicity over complexity. For example, use lexical search (BM25) instead of embeddings for tool discovery, and adopt a single compression path instead of offering a menu of options. LLMs are already quite smart, so adding more machinery in front of them often just adds complexity without guaranteed value. Build the simplest thing that works, then extend.
Use agents to optimize their own stack. Codex wrote a large portion of the API migration work that serves Codex, turning a multi-year rewrite into a few engineers' work over months. The inference team uses it to analyze traces and prototype kernels. When agents write code, you no longer need a language convenient for humans, so you can pick the language that runs most efficiently. Efficiency optimization is becoming a self-accelerating loop.
Optimize end-to-end. The inference team said every time they picked a single favorite technique to focus on, they later regretted it. Focusing on one part of the stack leads to under-investment in others. No single optimization is a breakthrough by itself. The real gains come from chaining many small optimizations together. They also learned to test with the same traffic shapes that production serves, because a change that looks good offline can backfire under real traffic.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
