Stateless Design at Ten‑Million QPS: Moving State Out of Compute for Elastic Scaling

During a Friday afternoon capacity‑expansion rehearsal, adding 20 machines in five minutes caused many users to be logged out and lose their shopping‑cart data because session information was stored in each machine’s local memory.

Where State Hides

A service is stateful when it keeps data that must survive across requests on the local instance. The article identifies four common categories: HttpSession: user ID, permissions, cart snapshot stored in memory.

Local caches such as ConcurrentHashMap or Caffeine that become de‑facto state when business logic assumes the data is always present.

Local files (uploaded images, temporary reports, multipart chunks) that disappear if the host crashes.

Long‑lived connections (WebSocket, IM, push channels) that are bound to a specific process.

When state is scattered across machines, load balancers lose freedom: they must route a user’s request back to the instance that holds the state, leading to sticky sessions, uneven load, failed scaling, and loss of sessions during shutdown.

The Cost of Being Stateful

Sticky sessions – a shackles

Enabling sticky sessions forces the load balancer to sacrifice even distribution to accommodate local state. This creates a cascade of problems: overloaded machines, ineffective scaling (new machines receive no traffic), and forced session termination when a machine is taken down.

Scaling dilemmas

During scale‑out, new instances lack any historic sessions, so they cannot serve existing users; during scale‑in, shutting down a machine discards all sessions, forcing users to re‑login.

Disaster recovery nightmare

If a machine crashes, all locally stored sessions, caches, and un‑persisted data vanish, causing abrupt logouts and data loss.

Release constraints

Rolling, blue‑green, or canary deployments rely on the ability to replace instances at any time. Stateful services break this assumption because restarting an instance drops its sessions.

Container incompatibility

In Kubernetes or serverless environments, pods are immutable and may be killed at any moment. Any state that lives on the container disappears like writing on sand.

Statelessness Defined

Stateless design does not eliminate state; it moves state out of the compute layer into dedicated storage, allowing any request to be handled by any instance with identical results.

The key criterion: a request sent to any instance must produce the same outcome without requiring a specific machine.

Four Paths to Externalize State

Different state types have different destinations.

Session state: centralized store or token

Two mainstream approaches:

Centralized session store : Move HttpSession to a shared Redis cluster. Frameworks like Spring Session redirect reads/writes to Redis, making the instance indifferent to which machine handles the request.

Tokenization : Encode user identity, permissions, and expiration into a JWT sent to the client. The server validates the token on each request, eliminating server‑side session storage.

Tokenization incurs drawbacks: revocation is hard (requires a blacklist), token size inflates HTTP headers, and sensitive data must not be placed in the token.

Business data: store in DB or distributed cache

Business data belongs in a database or distributed cache. Local caches should be read‑only, rebuildable, and discardable; they must never be relied upon as the sole source of truth.

File uploads: use object storage

Upload files directly to object storage services (e.g., S3, OSS). The compute instance only forwards data and never retains files locally, enabling any instance to handle uploads.

Long connections: separate connection handling

Introduce a gateway layer that maintains physical connections and a routing table in shared storage. Business services send messages to the gateway, which looks up the appropriate connection, keeping the business layer stateless.

What Statelessness Unlocks

True horizontal scaling – adding instances instantly adds capacity.

Unconstrained load balancing – any algorithm (round‑robin, least‑connections, consistent hashing) can be used.

Instant replacement of failed instances – no session loss.

Zero‑downtime deployments – rolling, blue‑green, canary releases work seamlessly.

Cloud‑native friendliness – aligns with immutable infrastructure, Deployments, and serverless execution.

Not a Silver Bullet

Externalizing state merely relocates the bottleneck to the storage layer. A Redis cluster that once served a few thousand sessions now becomes the critical shared state for millions of instances; its availability and performance become paramount.

Performance trade‑off: local memory reads are nanoseconds, remote Redis reads add a network round‑trip (milliseconds). At tens of millions of QPS, this latency multiplies into substantial load on the storage backend.

Mitigation strategies include layered caching (local cache in front of Redis) with strict discardability, or tokenization to avoid the extra hop at the cost of larger request payloads.

When Stateful Services Still Make Sense

Components that inherently require state—databases, message queues, Flink keyed state, ZooKeeper/etcd, game or IM servers—should remain stateful and be managed with StatefulSets or managed services. They are not candidates for forced statelessness.

Correct architecture: Deploy the stateless application layer with Deployments; run stateful back‑ends with StatefulSets or managed services, each optimized independently.

Performance Trade‑offs

Strong consistency demands remote reads; read‑heavy, eventually‑consistent workloads benefit from multi‑level caching. No universal solution exists—choices depend on workload characteristics.

Evolution Roadmap

The transition is incremental:

Start with a fully stateful service using in‑memory sessions and sticky routing.

Externalize sessions to Redis, removing sticky sessions and enabling free load balancing.

Optionally adopt JWT tokenization for scenarios that require zero server‑side session storage.

Finalize with a layered architecture: stateless compute (Deployments) plus a highly available, sharded storage layer (Redis clusters, partitioned databases, multi‑level caches).

At the ten‑million‑QPS scale, this layered model is the only viable architecture: the compute tier must be stateless to scale in seconds, while the storage tier must be engineered for high‑availability and horizontal scalability.

Final Thoughts

Stateless design does not eradicate state; it relocates state to a dedicated, shared storage layer, trading elastic compute for strong reliance on that storage’s availability, consistency, and performance.

Understanding which parts of your system still hold hidden local state is the first step toward a resilient, scalable architecture.