Layer‑4 vs Layer‑7 Load Balancing: A Complete Enterprise Architecture Guide

The claim “L4 is faster, L7 is more flexible” is only a starting point; the selection must answer protocol visibility, routing data, TLS termination, client‑IP preservation, health‑check depth, gray‑release capability, and where the bottleneck lies (connections, CPU, bandwidth, or backend dependencies).

1. Build the Correct Model

L4 load balancing operates at the transport layer, forwarding raw byte streams based on the TCP/UDP five‑tuple, connection state, or source‑IP hash. It cannot parse HTTP paths, headers, cookies, or gRPC methods. L7 parses application protocols and can route on Host, path, Header, Cookie, method, status code, etc., but adds complexity such as protocol parsing, TLS handling, retries, and application‑layer security.

First verify the inbound traffic protocol and TLS termination point. Do not infer the protocol from the port; capture packets and inspect the actual listening state.

ss -ltnp '( sport = :80 or sport = :443 or sport = :<frontend_port> )'
sudo tcpdump -ni any -c 20 'tcp port <frontend_port>'

For HTTP entry points, use curl to check response headers, protocol, and redirects. For non‑HTTP TCP services, use nc for a minimal connectivity check, which only proves the three‑way handshake and port reachability.

curl -vkI --max-time 5 https://<VIP>/<health_check_path>
nc -vz -w 3 <VIP> <frontend_port>

Check the HAProxy version, compile options, and current configuration because support for HTTP/2, TLS, Prometheus exporter, and runtime API varies between versions.

haproxy -vv
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl status haproxy --no-pager

2. L4: Fast, Stable Byte‑Stream Forwarding

L4 is suitable for database proxies, message queues, long‑lived game connections, TLS pass‑through, multi‑protocol entry points, and any scenario that needs end‑to‑end encryption. The trade‑off is the inability to route by URI or generate HTTP‑level error pages.

Minimal TCP configuration example (HAProxy 2.4+):

global
    log /dev/log local0
    maxconn <max_connections>

defaults
    mode tcp
    timeout connect 5s
    timeout client 60s
    timeout server 60s

frontend tcp_frontend
    bind :<frontend_port>
    default_backend tcp_backends

backend tcp_backends
    balance roundrobin
    server node1 <backend1_ip>:<port> check
    server node2 <backend2_ip>:<port> check

Before reloading, back up the configuration and run a syntax check. Reloads are usually smooth, but gray‑release nodes should be tested for connection migration, long‑connection handling, and file‑descriptor headroom.

sudo cp -a /etc/haproxy/haproxy.cfg /var/backups/haproxy.cfg.$(date +%Y%m%d_%H%M%S)
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

TCP health checks only verify connectivity; for databases or custom protocols, use protocol‑specific probes or dedicated health‑check endpoints.

backend tcp_backends
    option tcp-check
    default-server inter 2s fall 3 rise 2
    server node1 <backend1_ip>:<port> check
    server node2 <backend2_ip>:<port> check

Preserve source IP with the PROXY protocol; the backend must explicitly support it, otherwise connections fail.

backend tcp_backends
    server node1 <backend1_ip>:<port> check send-proxy
    server node2 <backend2_ip>:<port> check send-proxy

Observe connection states (ESTABLISHED, SYN‑RECV, TIME‑WAIT) to understand load beyond QPS.

ss -s
ss -tan state established '( sport = :<frontend_port> )' | wc -l
ss -tan state syn-recv '( sport = :<frontend_port> )' | wc -l

Source‑IP based stickiness can be configured, but NAT‑shared IPs may cause hotspots. Using hash-type consistent reduces remapping when scaling.

backend tcp_backends
    balance source
    hash-type consistent
    server node1 <backend1_ip>:<port> check
    server node2 <backend2_ip>:<port> check

3. L7: Routing and Governance via Request Semantics

L7 mode fits HTTP/HTTPS, REST, gRPC services. It enables path‑based routing, canary releases, pre‑authentication, compression, rate‑limiting, and observability, but requires explicit timeout, retry, request‑body limits, header trust boundaries, and TLS strategy.

Minimal HTTP configuration:

defaults
    mode http
    timeout connect 5s
    timeout client 60s
    timeout server 60s

frontend http_frontend
    bind :80
    acl is_api path_beg /api/
    use_backend api_backends if is_api
    default_backend web_backends

backend api_backends
    server api1 <backend1_ip>:<port> check

backend web_backends
    server web1 <backend2_ip>:<port> check

Validate routing with real Host headers and, for HTTPS, also test SNI, certificate chain, and redirects.

curl -sS -o /dev/null -w '%{http_code} %{remote_ip}
' \
    -H 'Host: <service_domain>' http://<VIP>/api/<health_path>
curl -sS -o /dev/null -w '%{http_code} %{remote_ip}
' \
    -H 'Host: <service_domain>' http://<VIP>/<health_path>

Terminate TLS at the front end; the clear text then travels to backends unless re‑encrypted. Audit certificate permissions, SNI coverage, protocol versions, and backend network isolation.

frontend https_frontend
    bind :443 ssl crt /etc/haproxy/certs/<cert_file>.pem alpn h2,http/1.1
    http-request set-header X-Forwarded-Proto https
    default_backend web_backends

Verify certificates before reload:

sudo test -r /etc/haproxy/certs/<cert_file>.pem
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
openssl s_client -connect <VIP>:443 -servername <service_domain> -brief

Pass client source information with option forwardfor and trust X‑Forwarded‑For only from controlled proxy networks.

frontend https_frontend
    bind :443 ssl crt /etc/haproxy/certs/<cert_file>.pem
    option forwardfor
    http-request set-header X-Forwarded-Proto https
    default_backend web_backends

HTTP health checks must hit a real health endpoint, expect a specific status code, and avoid write operations or user‑session dependence.

backend web_backends
    option httpchk GET /<health_path>
    http-check expect status 200
    default-server inter 2s fall 3 rise 2
    server web1 <backend1_ip>:<port> check
    server web2 <backend2_ip>:<port> check

Cookie stickiness supports stateful sessions but is generally discouraged in favor of shared storage or stateless tokens because it reduces load‑balancing effectiveness.

backend web_backends
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server web1 <backend1_ip>:<port> check cookie web1
    server web2 <backend2_ip>:<port> check cookie web2

Canary releases should use explicit headers or cookies; random traffic splits without version tags make regression analysis impossible.

frontend http_frontend
    bind :80
    acl canary hdr(X-Canary) -m str enabled
    use_backend api_canary if canary
    default_backend api_stable

backend api_stable
    server stable1 <backend1_ip>:<port> check

backend api_canary
    server canary1 <backend2_ip>:<port> check

Retries change request semantics; only enable for idempotent methods (GET/HEAD) and ensure business‑level idempotency keys for non‑idempotent operations.

backend api_stable
    option redispatch
    retries 2
    retry-on conn-failure empty-response 500 502 503 504
    server stable1 <backend1_ip>:<port> check

Before enabling retries, analyze logs to distinguish connection failures, timeouts, business errors, and proxy‑generated retries.

journalctl -u haproxy --since '30 min ago' --no-pager \
  | rg ' 5[0-9][0-9] |sC|SC|SD|SH' | tail -n 120

4. Observability and Load‑Testing Instead of Guesswork

If the Prometheus exporter is enabled, use the actual metric names it exposes. Both L4 and L7 should monitor sessions, queues, errors, and backend health; L7 adds request status, routing rule hits, and TLS errors.

frontend metrics
    bind 127.0.0.1:8404
    http-request use-service prometheus-exporter if { path /metrics }
    no log

Validate the exporter locally; never expose an unauthenticated metrics port to the public.

curl --fail --silent http://127.0.0.1:8404/metrics | head -n 30

Example PromQL queries (adjust job label to your HAProxy task name):

sum by (proxy,svname) (haproxy_frontend_current_sessions{job="<haproxy_job>"})
max by (proxy,svname) (haproxy_backend_current_queue{job="<haproxy_job>"})
sum(rate(haproxy_frontend_http_responses_total{job="<haproxy_job>",code=~"5.."}[5m])) /
    sum(rate(haproxy_frontend_http_responses_total{job="<haproxy_job>"}[5m]))

HTTP load testing should fix concurrency, duration, request body, keep‑alive, and TLS conditions, and must target only gray‑released entry points and read‑only paths.

wrk -t4 -c300 -d120s --latency \
    -H 'Host: <service_domain>' \
    https://<VIP>/<health_path>

For long‑lived TCP scenarios, do not replace TCP validation with a single HTTP QPS test; instead, verify connection counts, lifetimes, reconnections, and backend distribution.

for i in $(seq 1 50); do
    nc -z -w 3 <VIP> <frontend_port> &
 done
wait
ss -tan state established '( sport = :<frontend_port> )' | wc -l

5. Release, Switch, and Rollback

Before switching, back up the configuration, run syntax checks, record current backend status, and capture monitoring baselines. Gray‑release entry points must be validated for path, Header, TLS, health‑check, and source‑IP handling.

sudo cp -a /etc/haproxy/haproxy.cfg /var/backups/haproxy.cfg.$(date +%Y%m%d_%H%M%S)
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | head -n 30

After a single‑node gray reload, verify service status, listening ports, and that no write‑only health endpoints are exposed. For long‑connection workloads, ensure existing connections are retained or closed gracefully.

sudo systemctl reload haproxy
systemctl is-active haproxy
ss -ltnp '( sport = :80 or sport = :443 or sport = :<frontend_port> )'
curl --fail --silent --show-error https://<VIP>/<health_path> >/dev/null

If routing errors, TLS handshake failures, a surge of backend down marks, error‑rate spikes, or tail‑latency regressions occur, stop the rollout and revert to the last verified configuration. Perform another syntax check before rollback to avoid cascading failures.

sudo cp -a <backup_dir>/haproxy.cfg /etc/haproxy/haproxy.cfg
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

After rollback, re‑validate front‑end, back‑end, and critical error logs. Persistent issues may stem from backend services, network, DNS, certificates, or clients rather than the load‑balancing layer itself.

Enterprise selection is rarely a pure binary choice: an outer L4 layer can carry any TCP/TLS traffic and massive connections, while an inner L7 layer provides HTTP routing, gray releases, and security governance. The key is to document protocol boundaries, TLS termination, client‑IP handling, health‑check depth, timeout and retry semantics, monitoring metrics, and rollback procedures, then verify them with real traffic.