Supercharge Nginx Ingress Performance with Kernel and Config Tweaks

Overview

Nginx Ingress Controller implements the Kubernetes Ingress API using Nginx, a high‑performance gateway. To fully exploit its performance, both kernel and Nginx configuration parameters must be tuned.

Kernel Parameter Tuning

Adjust the following kernel settings to maximize Ingress performance under high load.

Increase the TCP listen backlog

The effective backlog size is the minimum of somaxconn and the Nginx backlog directive. Nginx defaults to 511, while the Ingress controller reads somaxconn (default 4096). Set it to a higher value, e.g., 65535.

sysctl -w net.core.somaxconn=65535

Expand the source port range

High concurrency can exhaust source ports. Expand net.ipv4.ip_local_port_range to cover the full range.

sysctl -w net.ipv4.ip_local_port_range="1024 65535"

Enable TIME_WAIT reuse

Allow sockets in TIME_WAIT to be reused, reducing port exhaustion. sysctl -w net.ipv4.tcp_tw_reuse=1 Also shorten FIN_WAIT2 and TIME_WAIT timers:

sysctl -w net.ipv4.tcp_fin_timeout=15
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Increase max TIME_WAIT connections

Raise net.ipv4.tcp_max_tw_buckets to handle more simultaneous TIME_WAIT sockets.

sysctl -w net.ipv4.tcp_max_tw_buckets=55000

Raise the maximum number of file handles

Nginx needs two file descriptors per request (client and upstream). Increase fs.file-max to allow more concurrent connections.

sysctl -w fs.file-max=1048576

Kernel tuning example

initContainers:
  - name: setsysctl
    image: busybox
    securityContext:
      privileged: true
    command:
    - sh
    - -c
    - |
      sysctl -w net.core.somaxconn=65535
      sysctl -w net.ipv4.ip_local_port_range="1024 65535"
      sysctl -w net.ipv4.tcp_max_tw_buckets=55000
      sysctl -w net.ipv4.tcp_tw_reuse=1
      sysctl -w fs.file-max=1048576
      sysctl -w net.ipv4.tcp_fin_timeout=15
      sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Application‑Layer Configuration Tuning

Increase keep‑alive request limit

Set keepalive_requests to a high value (e.g., 10000) so a single keep‑alive connection can serve many requests, reducing the number of TIME_WAIT sockets.

Increase keep‑alive idle connections for upstream

Adjust upstream‑keepalive‑connections (default 32) to a larger number such as 1000 to accommodate bursty traffic.

Gateway timeout settings

Fine‑tune the following timeouts to quickly drop stalled requests: proxy-connect-timeout: set to 3 seconds proxy-read-timeout: set to 3 seconds proxy-send-timeout: set to 3 seconds

Raise max worker connections

Increase max-worker-connections from the default 16384 to, for example, 65536.

Optimize retry mechanism

Disable upstream retry in the Ingress controller ( proxy-next-upstream="off") and limit retries in the edge Nginx using proxy_next_upstream_tries to avoid cascading retries.

Enable Brotli compression

Activate Brotli for better compression ratios than gzip. Set the following ConfigMap keys:

enable-brotli: "true"

brotli-level: "6"

(range 1‑11) brotli-types: "..." (list of MIME types)

ConfigMap example

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-controller

data:
  keep-alive-requests: "10000"
  upstream-keepalive-connections: "200"
  max-worker-connections: "65536"
  proxy-connect-timeout: "3"
  proxy-read-timeout: "3"
  proxy-send-timeout: "3"
  proxy-next-upstream: "off"
  enable-brotli: "true"
  brotli-level: "6"
  brotli-types: "text/xml image/svg+xml application/x-font-ttf image/vnd.microsoft.icon application/x-font-opentype application/json font/eot application/vnd.ms-fontobject application/javascript font/otf application/xml application/xhtml+xml text/javascript application/x-javascript text/plain application/x-font-truetype application/xml+rss image/x-icon font/opentype text/css image/x-win-bitmap"

References

Optimizing nginx‑ingress‑controller concurrency: https://cloud.tencent.com/developer/article/1537695

Nginx Ingress ConfigMap reference: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

Tuning NGINX for Performance: https://www.nginx.com/blog/tuning-nginx/

ngx_http_upstream_module documentation: http://nginx.org/en/docs/http/ngx_http_upstream_module.html