Why Images Fail to Load: Debugging TCP Listen/Accept Queues in Kubernetes Nginx Pods

Users reported that a large number of images could not be downloaded through a Kubernetes Ingress that forwards traffic to an Nginx deployment backed by an NFS‑mounted static file directory. The request chain is client → k8s ingress → nginx → NFS. Initial checks showed that the client could reach the Nginx pod IP but received no response, indicating a problem inside the pod.

No.1 Problem Description

The client’s SYN packets reach the Nginx container’s network interface, but Nginx never replies with a SYN‑ACK. iptables inside the container is empty, so the issue is not a firewall rule.

No.2 Guess

It is likely that the backend service handling the request is failing. A direct curl to the Nginx pod IP confirms that the service is unreachable.

No.3 Packet Capture

Using tcpdump inside the container’s network namespace on port 24568 shows repeated SYN packets arriving at the container, but no ACK is sent back, confirming that the SYN packets are being dropped after reaching the kernel.

No.4 Half‑Connection and Full‑Connection Queues

Linux creates two queues for a listening socket: syn queue: holds half‑open connections (SYN received, ACK not yet sent). accept queue: holds fully established connections waiting for the application to call accept().

Only after accept() is called does the connection reach the application layer.

No.5 listen vs accept

All server programs eventually call the listen() system call, which sets the size of both queues, and then repeatedly call accept() to pull connections from the accept queue.

No.6 Linux backlog

The listen() call takes a backlog argument. Modern kernels use this value for the accept queue size, while the SYN queue size is limited by /proc/sys/net/ipv4/tcp_max_syn_backlog (if syncookies are disabled). The effective maximum accept queue size is min(backlog, net.core.somaxconn).

No.7 Queue Overflow

When either queue is full, the kernel increments the ListenOverflows and ListenDrops counters and silently drops new SYN packets.

No.8 Back to the Issue

Running netstat -s inside the container shows large numbers for "listen overflows" and "listen drops", indicating that the accept queue is saturated. Checking the socket with ss -lnt shows Recv‑Q 129 and Send‑Q 128, meaning the accept queue limit is 128.

No.9 Default somaxconn is Small

Although the host node’s net.core.somaxconn is 32768, each network namespace starts with a default value of 128 (defined by SOMAXCONN 128 in the kernel source). This explains the low limit observed inside the pod.

No.10 Raising somaxconn

Three methods are presented:

Use the Kubernetes sysctls feature in the pod spec.

Set the value from an initContainer with privileged access.

Deploy the CNI tuning plugin to apply the sysctl cluster‑wide.

No.11 Nginx Backlog

Even after raising net.core.somaxconn, Nginx still reports a backlog of 511 because Nginx’s own listen directive defaults to 511 on Linux. The backlog parameter must be set explicitly in nginx.conf, e.g. listen 80 default backlog=1024;.

No.12 Takeaways

For high‑traffic services running in Kubernetes, adjust both the kernel net.core.somaxconn sysctl and the Nginx backlog setting to prevent accept‑queue overflow and lost SYN packets.

No.13 References

Using sysctls in a Kubernetes Cluster – https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/

SYN packet handling in the wild – https://blog.cloudflare.com/syn-packet-handling-in-the-wild/