Designing an Agent Gateway: Bridging Business Logic and Protocol Infrastructure

Why Traditional API Gateways Fall Short

Traditional API gateways assume stateless requests, performing independent authentication, routing, and rate‑limiting per HTTP request and discarding context afterward. In Agentic workflows, most processing nodes maintain state, such as the MCP protocol’s persistent bidirectional JSON‑RPC streams (STDIO or WebSocket) that keep context for the entire session.

Conflict Between Stateless Infrastructure and Stateful Protocols

Applying a stateless infrastructure layer over a stateful protocol creates three main problems:

Connection overhead : The older HTTP + SSE‑based MCP requires each tool session to open multiple TCP connections, causing the number of connections to grow linearly with the number of tools.

In‑session fan‑out aggregation : Requests like tools/list must concurrently fan out to N backend MCP servers, aggregate and deduplicate results, and return a unified view while preserving the JSON‑RPC context.

Server‑push routing : SSE events are pushed by the server, so the gateway must be session‑aware to correctly multiplex streams back to the appropriate client connections.

MCP and A2A

MCP solves the vertical integration problem—how an Agent calls a tool—using JSON‑RPC 2.0 where the client is the Agent and the server provides the tool. A2A solves the horizontal integration problem—how an Agent delegates to another Agent. The two layers are independent; stuffing Agent‑to‑Agent calls into MCP routing breaks session semantics and loses A2A’s task‑lifecycle management.

A2A introduces three state‑management layers: session‑level context, Agent‑level internal state, and task‑level persistence via a TaskStore. Its core primitive is the Agent Card, a JSON capability declaration that lists supported skills, input/output schemas, and security preconditions. Runtime discovery and delegation rely on this declaration.

Engineering Challenges in Distributed MCP

Session fan‑out is a pain point: the gateway must issue N concurrent requests to backend servers, wait, aggregate, deduplicate, and return a single view while keeping the same JSON‑RPC session. Without a gateway, this logic would be scattered across each Agent’s initialization code, becoming harder to maintain as the number of MCP servers grows.

Connection state management evolves with the MCP spec. Streamable HTTP transport solves horizontal scaling: short‑lived operations return immediate responses, while long‑running tasks upgrade to SSE streams without requiring session affinity. Backend MCP servers can sit behind a regular load balancer. However, the client still needs connection reuse, timeout retries, and idempotent handling of partial failures—responsibilities that remain in the gateway layer.

Serialization overhead becomes noticeable at high throughput because JSON‑RPC 2.0 adds extra cost compared with binary formats.

Context explosion occurs as workflows lengthen, since LLM context windows are limited resources.

Security Considerations

Agent security differs from traditional application security: the attack surface extends from code execution paths to the reasoning process. Prompt injection is a control‑layer attack, not an input‑validation issue; malicious commands are hidden in data consumed by the Agent and executed during LLM inference.

Indirect prompt injection can be more covert: an Agent reads an email or webpage containing a disguised command, parses it as a task, and executes it without ever accessing the Agent’s system directly. Once an Agent is authorized to read external content, the boundary between data and control layers disappears.

The gateway provides an execution boundary outside the reasoning process. Using CEL‑based RBAC policies, the gateway can translate the capability limits declared in an Agent Card into enforceable rules, intercepting calls such as email.send_to_external before they reach the MCP server. This interception occurs at the JSON‑RPC payload level, not merely at the HTTP request level, requiring the gateway to understand the semantics of tools/call, resolve the tool name and parameters, and apply the Agent’s authorization policy.

Agentic DoS (ADoS) is another attack vector: a hijacked Agent repeatedly triggers tool calls, exhausting API quotas or compute resources without generating high HTTP QPS. Traditional request‑level rate limiting is ineffective; the gateway must enforce throttling per Agent at the tool‑call level, decoupled from HTTP request counting.

Runtime risk accumulation is also a concern: individual tool calls may appear harmless, but a series can create data‑exfiltration paths. The security layer should maintain a time‑decaying session‑level risk score, perform quick heuristic scans, forward suspicious tool results to an LLM‑assisted classifier for deep analysis, and trigger progressive responses (slow down, require human approval, then abort the session) once a threshold is crossed.

Core Responsibilities of an Agent Gateway

The Agent gateway manages the entire workflow lifecycle.

Agent orchestration

Stateful workflow execution

Tool routing and permission management

Agent‑to‑Agent communication

Observability and traceability

Human approval processes

Memory and session management

Barrier and execution policies

Retry handling and fault recovery

It transforms the model from “single API call” to “hosted AI system.” Without a gateway, each team builds its own orchestration logic, which becomes unmaintainable as workflows grow in complexity, scale across dozens of teams and hundreds of workflows, and demand memory, permissions, state transitions, retries, and audit capabilities.

Observability for Agent Workflows

Traditional APM can report request latency and HTTP status but cannot reveal that an Agent generated hallucinated output in a second LLM call or that a tool was invoked an extra two times because of a prior response. Agent failures are often silent until users notice incorrect results.

Effective Agent observability should capture structured spans such as:

Tool‑call Span : tool name, parameters (hashed after sensitive field redaction), result summary, latency, error code

Reasoning Span : full LLM prompt (including system prompt version), completion, token usage distribution, model version

State Transition Span : workflow state changes and trigger conditions

Memory Span : memory read/write operations, cache hit rate, context window utilization

Agent Handoff Span : A2A delegation details—Agent Card version, capability match, delegation chain path

Conclusion

The necessity of an Agent gateway stems from a concrete technical contradiction: MCP and A2A are stateful, bidirectional, long‑lived protocols, whereas traditional gateways are stateless, unidirectional, and discard connections after each request. Reconciling these paradigms requires a dedicated, state‑aware gateway layer.