How Yelp Achieved Zero‑Downtime HAProxy Reloads Using Linux qdisc

Yelp’s infrastructure team aims for near‑zero downtime, meaning the website must respond as quickly as possible when users access www.yelp.com. One method is using HAProxy load balancing to keep the site responsive, both externally and internally, as part of their SmartStack architecture.

HAProxy Reload Causes Packet Loss

HAProxy version 1.5.11 does not support a true zero‑downtime restart. It performs a fast reload: a new HAProxy instance starts and tries to bind to the same ports using SO_REUSEPORT, while signaling the old instance to shut down. During the brief moment when both processes are bound, the Linux kernel may drop packets, especially SYN packets, resulting in RST responses and lost connections.

The HAProxy maintainer, Willy Tarreau, recommends dropping SYN packets during reloads and relying on TCP’s automatic retransmission. However, RFC 6298 mandates a 1‑second initial SYN timeout, and many clients (including mobile devices) retry after 200 ms to 3 s, causing noticeable delays.

Making HAProxy Reloads Lossless

To avoid the delay, Yelp adopted Willy’s suggestion but improved it by delaying SYN packets only until the reload completes. This is achieved with Linux queueing disciplines (qdiscs), specifically the plug qdisc, combined with iptables to mark incoming SYN packets.

Using tc (Linux traffic control) and the libnlutils library, Yelp creates a filter‑based rule that redirects SYN packets to a special queue during a reload. The plug qdisc holds these packets without dropping them, and once the new HAProxy instance is ready, the packets are released.

tc : Linux traffic control tool that allows creation of filter‑based queuing rules.

iptables : Linux utility for packet filtering and NAT, used here to identify and mark incoming SYN packets.

The SmartStack client connects to the loopback interface, sending a request to HAProxy. HAProxy converts the incoming packet to an outgoing one, allowing the loopback interface to host the queue shown in Figure 1.

In Yelp’s implementation, the standard prio qdisc is used with its fourth channel – the plug qdisc – which retains packets instead of discarding them. Combined with an iptables rule, this setup redirects SYN packets during the HAProxy reload, then releases them once the reload finishes.

The solution is packaged into a script called qdisc_tool protect <normal HAProxy reload command>, which can be run on typical Linux distributions such as Ubuntu Trusty. If the system lacks nl-qdisc-add but runs a kernel 3.4+ with netlink support, the plug qdisc can be manipulated manually.

By employing this technique, Yelp’s infrastructure can protect HAProxy reloads from packet loss, achieving near‑zero‑downtime updates without the extra latency introduced by dropping SYN packets.