Nginx Unified Gateway Deep Dive: Multi‑Domain, Dynamic Routing, and Ten‑Million Concurrency Load Balancing

Why Enterprises Move to a Unified Gateway

Early Nginx deployments expose one or a few servers on ports 80/443 and route by domain or path, adding a new server or location block for each service. This works for small projects but quickly fails when domain count, service instances, traffic models, gray‑release, TLS handshakes, rate limiting, and operational complexity grow.

What Nginx Is Good At and What It Isn’t

Strengths

L7 reverse proxy, TLS termination, Host/Path/Header routing, connection reuse, static file serving, basic rate limiting, and gateway‑level logging.

Limitations

Complex business orchestration, state persistence, service‑registry responsibilities, fine‑grained policy editing, and large‑scale configuration collaboration belong to a control plane.

Core Reasons Behind "Ten‑Million Concurrency"

Most articles conflate the term with a single‑machine myth. In reality, "ten‑million" refers to cluster‑wide concurrent connections, which depend on file‑descriptor limits, worker connection limits, NIC capacity, TLS CPU, and upstream pool resources.

total_capacity = min(
    file_descriptor_limit,
    worker_connection_limit,
    nic_bandwidth_limit,
    tls_cpu_limit,
    upstream_pool_limit,
    kernel_socket_queue_limit
)

Master/Worker Model and Event‑Driven I/O

The master process reads configuration and performs graceful reloads, while each worker runs a single‑threaded event loop using epoll. This avoids thread‑switch overhead and is ideal for I/O‑bound workloads.

Dynamic Routing Strategies

Routing Dimensions

Host, Path, Header, Cookie, Method, Tenant, Geo.

Prefer simple Host+Path as the primary route, keep gray‑release logic explicit, and let the control plane generate routing rules.

Dynamic Upstream Implementations

DNS‑based resolution – simple but coarse.

Config generation + smooth reload – clear audit trail, but reload cost can be high.

OpenResty balancer_by_lua – avoids frequent reloads, suitable for high‑frequency instance changes.

Ingress controller / gateway controller – delegates discovery and config generation to Kubernetes.

Load‑Balancing Algorithms and Their Trade‑offs

Round‑robin – simple, may skew under long connections. least_conn – good for long‑lived connections, less optimal for short spikes.

IP hash – easy, but mapping changes with scaling.

Consistent hash – stable for tenant or cache routing, but key design is critical.

Random two least_conn – mitigates hot spots, needs thorough stress testing.

Rate Limiting, Circuit Breaking, and Degradation

Three‑layer rate limiting (IP, API, tenant) protects the system without denying users outright. Circuit breaking monitors 5xx ratio, upstream timeouts, response‑time spikes, and connection failures, using shared memory in OpenResty to open a breaker when thresholds are crossed.

access_by_lua_block {
    local breaker = ngx.shared.breaker_dict
    local key = "svc:order-service"
    local state = breaker:get(key)
    if state == "open" then
        ngx.status = 503
        ngx.header["Content-Type"] = "application/json"
        ngx.say('{"code":"UPSTREAM_DEGRADED","message":"service temporarily unavailable"}')
        return ngx.exit(503)
    end
}

log_by_lua_block {
    local breaker = ngx.shared.breaker_dict
    local key = "svc:order-service"
    if ngx.var.upstream_status and ngx.var.upstream_status:find("5") then
        local fails = breaker:incr(key..":fails", 1, 0, 10)
        if fails and fails >= 50 then
            breaker:set(key, "open", 30)
        end
    end
}

Production‑Grade Configuration Skeleton

worker_processes auto;
worker_rlimit_nofile 200000;

events {
    use epoll;
    worker_connections 65535;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    server_names_hash_bucket_size 128;
    server_names_hash_max_size 8192;
    client_max_body_size 20m;
    log_format gateway_json escape=json '{"time":"$time_iso8601","request_id":"$request_id","remote_addr":"$remote_addr","host":"$host","method":"$request_method","uri":"$uri","status":$status,"request_time":$request_time,"upstream_addr":"$upstream_addr","upstream_status":"$upstream_status","upstream_response_time":"$upstream_response_time","bytes_sent":$bytes_sent,"http_referer":"$http_referer","http_user_agent":"$http_user_agent"}';
    access_log /var/log/nginx/access.log gateway_json buffer=512k flush=1s;
    error_log /var/log/nginx/error.log warn;
    # … upstreams, maps, and server blocks …
}

The configuration emphasizes layered separation (public, API, admin, static), explicit timeouts, structured logging, and per‑layer rate limiting.

Capacity Planning and System Parameters

Estimate per‑node capacity as worker_processes * worker_connections / 2 because each client connection also consumes an upstream connection. Adjust with TLS CPU, long‑connection idle, keepalive, logging I/O, and kernel socket/backlog settings. Real capacity must be validated by load testing.

fs.file-max = 1000000
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time = 600

Kubernetes Deployment Considerations

When running Nginx as a Deployment, set terminationGracePeriodSeconds, a preStop hook that sends nginx -s quit, and a readiness probe that checks the actual configuration health. Decouple configuration changes from image updates to keep rollouts safe.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-nginx
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 90
      containers:
      - name: nginx
        image: nginx:stable
        ports:
        - containerPort: 80
        - containerPort: 443
        readinessProbe:
          httpGet:
            path: /healthz
            port: 18080
          initialDelaySeconds: 5
          periodSeconds: 5
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "nginx -s quit; sleep 20"]
        resources:
          requests:
            cpu: "2"
            memory: "2Gi"
          limits:
            cpu: "4"
            memory: "4Gi"

Common Pitfalls

Incorrect trailing slash in proxy_pass changes URI rewriting.

Default timeouts that are too long or too short; split by traffic type.

Unbuffered synchronous logging causing I/O pressure.

Using round‑robin for long‑lived connections leads to imbalance.

Heavy reliance on regex routing hurts performance and readability.

Frequent reload indicates a flawed dynamic routing design.

Inconsistent certificate distribution across nodes.

Monitoring only the gateway, not upstream health.

Embedding complex business logic in Lua scripts.

Missing audit for gray‑release rules.

Forgetting to disable buffering for streaming interfaces.

Only testing normal traffic; neglecting failure scenarios such as upstream timeouts, TLS handshake storms, rate‑limit spikes, node loss, and bad deployments.

Pre‑Launch Checklist

All configs generated from a unified model and pass nginx -t.

No domain or route conflicts; audit records exist.

Capacity evaluated for concurrent connections, QPS, and TLS handshake peaks.

Rate‑limit and circuit‑breaker thresholds validated by stress tests.

Structured logging and request IDs propagate through the entire trace.

Three‑layer observability (node health, request path health, upstream health) with actionable alerts.

Fault‑injection and rollback drills for node loss, upstream degradation, certificate rollout failures, and config errors.

Conclusion

The true value of a unified Nginx gateway lies not in clever proxy_pass tricks but in building an engineering system that provides clear data‑plane handling, independent control‑plane governance, robust high‑concurrency protection, and end‑to‑end observability, audit, and rollback capabilities.