Why LLMs Are Undermining 20‑Year‑Old Stateless Web Architecture

Traditional Stateless Web Architecture

Web services have been built on a 20‑year‑old assumption: application state lives in a database while compute nodes are stateless. Scaling the system therefore requires vertical scaling of the database (using larger machines), whereas application servers can be horizontally scaled by adding more instances behind a load balancer.

How LLMs Break the Assumption

Long‑running work : AI agents may execute tasks that last minutes, turning a single request into an asynchronous, long‑running process.

Stateful compute : Agents keep conversational context, perform multiple tool calls, and rely on in‑memory state that is not stored in the database.

Bidirectional interaction : Users want to observe the agent’s reasoning, interrupt it, and guide it, creating a dialogue with the process rather than a simple stateless API call.

Persistent Execution and Its Limits

Frameworks such as Temporal, Ingest, and Restate provide durable, resilient execution for long‑running workflows. They give the workflow persistence but still treat the underlying transport as stateless, so they do not solve the interaction problem.

The Routing Problem

With HTTP, a load balancer, stateless servers, and a database, there is no way to route a request to a specific process; routing can only target the database. When a client needs to communicate with a process running inside a persistent‑execution framework, the same limitation appears, leading many to poll the database for updates. Polling incurs latency, adds load to the database, wastes requests, and yields a poor streaming experience.

Missing Routing Primitive

What is lacking is a routable transport name that is not tied to a particular server instance. Ideally one could say, “send this message to whoever is generating output for workflow X,” without knowing the machine, replica, or process.

Potential Solutions

WebSocket?

WebSocket provides a long‑lived, bidirectional connection between client and server, solving the routing problem for the duration of the connection. However, it is a connection, not an address; if the connection drops, the address is lost and the client cannot reconnect to the same process.

Publish/Subscribe Channels

Pub/sub inverts ownership: the server process and client are not directly addressable, but the transport layer is. Both sides connect to a named channel, which acts as the address. The channel is persistent, allowing disconnections and reconnections without losing data or routing capability.

In a workflow example, a persistent step connects to a pub/sub channel named after the workflow ID. The client also connects to that channel to receive updates and can send interrupt or guidance messages. Because both the workflow and the channel persist, they can reconnect after a failure and continue communication.

Why This Matters for LLMs

LLM responses are nondeterministic and costly. Retries that are cheap for deterministic APIs become expensive when each token incurs a cost. Losing a connection means losing paid tokens, and re‑sending them through the database is undesirable. Thus LLMs make the shortcomings of the classic HTTP + stateless + database stack more painful.

Conclusion

Stateless web architecture is not wrong, but it is ill‑suited for agent‑type applications that require long‑running, stateful, interactive processes. A new architecture is needed that introduces routing primitives capable of addressing processes directly, combined with persistent execution and pub/sub channels to handle durability, resilience, and bidirectional communication.