Why HAProxy May Drop Requests When a Backend Fails – A Packet‑Capture Study

Motivated by questions about load‑balancing reliability, the author performed packet‑capture experiments on HAProxy and Nginx to observe how each handles a failing backend server.

HAProxy Experiment

The HAProxy configuration uses inter 20000 (20 seconds) for health checks to make the detection interval obvious.

listen test9090
    bind 127.0.0.1:9090
    mode tcp
    server localhost90 127.0.0.1:90 check inter 20000
    server localhost91 127.0.0.1:91 check inter 20000

Two Nginx instances listen on ports 90 and 91, serving a simple index.html from /var/www/html:

server {
    listen 90;
    listen 91;
    location / {
        root /var/www/html;
    }
}

Using curl 127.0.0.1:9090 and running tcpdump -i lo -nn 'port 90' and tcpdump -i lo -nn 'port 91' in separate terminals, the author verified that HAProxy distributes traffic to both backends.

Packet captures confirmed that both ports 90 and 91 were receiving traffic.

Observing HAProxy Health Checks

Because of the 20‑second interval, HAProxy performs health probes only when there is no client traffic. The capture shows a probe every 20 seconds, separate from request handling.

Simulating a Backend Failure

Removing the listen 91 directive from the Nginx configuration and reloading simulates a failure of the second backend. HAProxy continues to forward some requests to the dead server until three consecutive failed health checks occur, which can take up to 60 seconds (3 × 20 s). During this window, any incoming requests to the failed backend are lost.

Nginx Load‑Balancing Experiment

The Nginx reverse‑proxy configuration uses an upstream block with weighted servers:

upstream backend {
    server 127.0.0.1:90 weight=1 max_fails=3 fail_timeout=30;
    server 127.0.0.1:91 weight=5 max_fails=3 fail_timeout=30;
}

server {
    listen 9090;
    location / {
        proxy_pass http://backend;
    }
}

When the listen 91 line is removed and Nginx is reloaded, the front‑end continues to serve requests without error. Packet captures show traffic still arriving on port 90, and Nginx automatically retries the healthy server when the failed one cannot provide data.

Key Takeaways

HAProxy performs continuous health checks; if a backend fails, requests that arrive during the detection window may be lost, and HAProxy routes traffic to only one backend at a time.

Nginx does not run proactive health checks when idle. It forwards each request to a backend, and if the chosen server does not respond, it retries the other server, effectively avoiding request loss as long as overall concurrency can handle the extra round‑trip.

Consequently, using HAProxy as a front‑end load balancer can impact users during backend maintenance under high load, whereas Nginx tolerates backend failures more gracefully, provided the system can sustain the required concurrency.