From Short Connections to Connection Pools: Managing Millions of QPS

Why Short Connections Were Sufficient at First

Short connections create a new TCP connection for each request and close it immediately after the response. This stateless approach is simple, requires no connection state maintenance, and was adequate when QPS was only hundreds or thousands because the handshake cost was negligible.

The core value of short connections is simplicity, not performance. When the request rate is low, the extra RTT for the three‑way handshake is hidden by abundant resources, so adding connection reuse would only add unnecessary complexity.

Cost of Short Connections in High‑Frequency Scenarios

Three main costs become significant as QPS rises:

Handshake latency: Each request must complete a full RTT, which can be several milliseconds across data centers, effectively doubling latency for latency‑sensitive services.

TIME_WAIT accumulation: The side that actively closes the connection enters TIME_WAIT for roughly twice the maximum segment lifetime. At tens of thousands of new connections per second, thousands of sockets remain in TIME_WAIT, exhausting local port ranges and causing "no available port" errors.

CPU and kernel overhead: Creating and destroying sockets incurs system‑call and context‑switch costs. Multiplied by millions of requests per second, this becomes a non‑trivial load on the kernel.

These costs are proportional, not absolute: they are noise at 100 K QPS but dominate at 10 M QPS, turning connection management into a primary performance factor.

Step 1: Reuse Connections with Long‑Lived Connections

The natural first evolution is to keep connections alive after the initial handshake—so‑called long connections or persistent connections. HTTP/1.1 Keep‑Alive is a classic example.

Long connections eliminate the per‑request handshake cost, but they introduce new challenges: idle connections may be silently dropped by firewalls or load balancers, remote services may restart, and idle connections need health‑checking.

Application‑level heartbeats (seconds‑level intervals) are recommended over relying on OS‑level TCP Keep‑Alive, which defaults to two‑hour intervals and is far too slow for high‑frequency services.

Step 2: From One Long Connection to a Connection Pool

A single long connection becomes a bottleneck when its throughput cannot satisfy concurrent requests. Pre‑creating a pool of long connections allows multiple requests to be processed in parallel.

The pool balances two opposing goals:

Capacity: Enough idle connections to avoid waiting for a new handshake during traffic spikes.

Resource usage: Avoid keeping too many idle connections that waste memory and file descriptors.

When a request needs a connection, it borrows one from the pool; if none are free, it either blocks or creates a temporary connection up to the configured maximum. The pool must also handle reclamation of idle connections and health‑check failed connections before reuse.

Core Parameters of a Connection Pool

Minimum idle connections: Guarantees ready‑to‑use connections during low traffic.

Maximum connections: Caps resource consumption and protects downstream services.

Idle timeout: Closes connections that have been idle beyond a threshold, allowing the pool to shrink during troughs.

Health check: Validates connections before borrowing or returning them, discarding broken ones.

These parameters must be tuned to the specific traffic pattern, number of downstream instances, and per‑connection throughput; there is no universal setting.

Qualitative Change at Tens of Millions of QPS

At the 10 M QPS scale, new problems appear:

Connection explosion: In a mesh of 100 instances calling 100 downstream instances, each pair may maintain its own pool, resulting in hundreds of thousands of connections.

Load imbalance: Uneven distribution of connections across downstream instances creates hotspots that are amplified at massive scale.

Failure avalanche: When a downstream instance fails, many pooled connections become invalid; a sudden surge of reconnection attempts can overwhelm the recovering instance.

These issues require moving from per‑service pool tuning to global connection governance, including protocol‑level multiplexing, connection‑level load balancing, and coordinated fault‑tolerance mechanisms.

Common Pitfalls

Pool‑to‑scale mismatch: Adding new downstream instances does not automatically redistribute existing long connections, leading to traffic sticking to old instances.

Connection leaks: Failure to return borrowed connections gradually exhausts the pool, causing latency spikes and eventual deadlock under high load.

Blindly copying configurations: Pool sizes tuned for a different system may be far from optimal; parameters must be derived from own load‑testing data.

These pitfalls are hidden in the connection layer, often invisible to business‑level metrics, yet they can cripple a service despite green health indicators.

Choosing the Right Strategy at Different Scales

A decision tree guides the selection:

~100 K QPS: Short connections are still advantageous due to their simplicity; introducing a pool adds unnecessary complexity.

~1 M QPS: Adopt long connections with a well‑tuned pool; focus on capacity, idle reclamation, and health checks.

~10 M QPS: Go beyond a single pool: employ protocol multiplexing, distributed load balancing, and global connection topology management.

The evolution is not a rejection of earlier solutions but a progressive incorporation of new constraints as scale grows.