Layer 4 vs Layer 7 Load Balancing: A Practical Decision Framework for Enterprise Architects

1. Establish the Correct Model

Layer 4 load balancing operates at the transport layer: the proxy forwards byte streams based on TCP/UDP 5-tuple, connection state, and source IP hash. It does not understand HTTP paths, headers, cookies, or gRPC methods, and should not attempt routing based on these fields. Layer 7 load balancing parses the application protocol and can apply policies based on host, path, header, cookie, request method, and status code, but introduces complexity around protocol parsing, TLS, retries, and application-layer security.

First confirm the ingress traffic protocol and TLS termination location. Do not guess the protocol by port number; packet captures and actual listening state are the evidence.

ss -ltnp '( sport = :80 or sport = :443 or sport = :<frontend_port> )'
sudo tcpdump -ni any -c 20 'tcp port <frontend_port>'

For HTTP ingress, verify response headers, protocol, and redirects with curl. The command only reads the health check path, but should still use a path that does not write business data.

curl -vkI --max-time 5 https://<VIP>/<health_check_URL>

For non-HTTP TCP services, use nc for minimal connectivity checks. A successful connection only proves the three-way handshake and port reachability, not authentication, business protocol, or backend dependency health. nc -vz -w 3 <VIP> <frontend_port> Check HAProxy version, compiled features, and active configuration. Different versions vary in support for HTTP/2, TLS, Prometheus exporter, and runtime API.

haproxy -vv
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl status haproxy --no-pager

Before selection, write requirements as testable items rather than vague descriptions like "high performance" or "powerful features". The following decision factors should be populated with actual values from service owners, traffic data, and security requirements.

Protocol: Layer 4 favors arbitrary TCP/UDP or no decryption; Layer 7 favors HTTP/HTTPS, gRPC, or need to understand requests.

Routing Conditions: Layer 4 uses IP, port, connection hash; Layer 7 uses domain, path, header, cookie, method.

TLS: Layer 4 prefers end-to-end passthrough or backend termination; Layer 7 prefers centralized ingress termination, certificate governance, WAF.

Session: Layer 4 connection-level stickiness suffices; Layer 7 needs cookie/request-level stickiness, canary identifiers.

Health Checks: Layer 4 checks TCP reachability; Layer 7 checks HTTP status codes, response content, auth paths.

Risk: Layer 4 passthrough is simple but offers less application observability; Layer 7 policies are powerful but retries/timeouts can change semantics.

2. Layer 4: Suitable for Fast, Stable Byte Stream Delivery to Backends

Layer 4 is not a "low-end Layer 7". Database proxies, message queues, game long connections, TLS passthrough, multi-protocol ingress, and end-to-end encryption scenarios may all be better suited to TCP mode. The trade-off is inability to route by URI and inability to generate HTTP-level error pages or redirects.

A minimal TCP frontend/backend configuration follows. balance roundrobin schedules at the connection level; with long connections, connection count balance does not equal request load balance.

global
    log /dev/log local0
    maxconn <proxy_max_connections>

defaults
    mode tcp
    timeout connect 5s
    timeout client  60s
    timeout server  60s

frontend tcp_frontend
    bind :<frontend_port>
    default_backend tcp_backends

backend tcp_backends
    balance roundrobin
    server node1 <backend_addr1>:<port> check
    server node2 <backend_addr2>:<port> check

Configuration must pass syntax check before loading. Reload is usually graceful, but still verify connection migration, long connections, and file descriptor headroom on canary nodes.

sudo cp -a /etc/haproxy/haproxy.cfg /var/backups/haproxy.cfg.$(date +%Y%m%d_%H%M%S)
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

TCP health check only verifies that a connection can be established. For databases or custom protocols, an open port does not mean the service is readable/writable; deep checks require real protocol probes or application-level dedicated endpoints.

backend tcp_backends
    option tcp-check
    default-server inter 2s fall 3 rise 2
    server node1 <backend_addr1>:<port> check
    server node2 <backend_addr2>:<port> check

Source IP preservation is often overlooked in Layer 4 designs. Transparent forwarding, DNAT, SNAT, and PROXY protocol behave differently; enabling send-proxy requires explicit backend support for PROXY protocol, otherwise connections fail immediately.

backend tcp_backends
    server node1 <backend_addr1>:<port> check send-proxy
    server node2 <backend_addr2>:<port> check send-proxy

Before changes, verify backend listeners support PROXY protocol. Different services have different enablement parameters; consult the specific service version documentation. The following commands only check port and current process, not protocol support.

ss -ltnp '( sport = :<port> )'
sudo tcpdump -ni any -c 5 'tcp port <port>'

Connection counts, queues, and error rates help identify Layer 4 pressure points. HAProxy admin socket commands are read-only on running processes, but the socket path must match the actual global stats socket configuration.

echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | head -n 20
echo 'show info' | sudo socat stdio /run/haproxy/admin.sock

When long connections are numerous, observing connection lifecycle is more important than just QPS. TIME-WAIT, SYN-RECV, ESTABLISHED must be interpreted alongside client behavior, timeout parameters, and backend capacity.

ss -s
ss -tan state established '( sport = :<frontend_port> )' | wc -l
ss -tan state syn-recv '( sport = :<frontend_port> )' | wc -l

Layer 4 can also use source IP for connection stickiness, but NAT scenarios where many clients share an egress IP can cause hotspots. hash-type consistent helps reduce remapping during backend scaling, but does not eliminate inherent traffic skew.

backend tcp_backends
    balance source
    hash-type consistent
    server node1 <backend_addr1>:<port> check
    server node2 <backend_addr2>:<port> check

During validation, examine real source IPs, backend selection, and errors from load test machines or log platforms, not just configuration inference. The following command only initiates non-business-write TCP connectivity tests.

for host in <load_test_IP1> <load_test_IP2>; do
    printf 'Initiate connectivity test from %s to <VIP>:<frontend_port> and record backend logs.
' "$host"
done

3. Layer 7: Trading Request Semantics for Routing and Governance Capabilities

Layer 7 mode suits HTTP/HTTPS, REST, gRPC, and other services where the request can be understood. It supports path-based routing, canary releases, authentication fronting, unified compression, rate limiting, and observability, but requires explicit timeouts, retries, request body limits, header trust boundaries, and TLS policies.

A minimal HTTP configuration lets HAProxy parse requests and route different paths to different backends. ACL order and default backend must be tested to avoid new paths accidentally falling into the wrong service.

defaults
    mode http
    timeout connect 5s
    timeout client  60s
    timeout server  60s

frontend http_frontend
    bind :80
    acl is_api path_beg /api/
    use_backend api_backends if is_api
    default_backend web_backends

backend api_backends
    server api1 <backend_addr1>:<port> check

backend web_backends
    server web1 <backend_addr2>:<port> check

Before going live, validate routing with different paths. Requests must include the real Host; if the system uses HTTPS, tests should also cover SNI, certificate chain, and redirects.

curl -sS -o /dev/null -w '%{http_code} %{remote_ip}
' \
    -H 'Host: <business_domain>' http://<VIP>/api/<health_check_URL>
curl -sS -o /dev/null -w '%{http_code} %{remote_ip}
' \
    -H 'Host: <business_domain>' http://<VIP>/<health_check_URL>

Layer 7 TLS termination centralizes certificate management, but plaintext appears on the proxy-to-backend link unless re-encrypted. Certificate private key permissions, SNI coverage, protocol versions, and backend network isolation must be audited together.

frontend https_frontend
    bind :443 ssl crt /etc/haproxy/certs/<cert_file>.pem alpn h2,http/1.1
    http-request set-header X-Forwarded-Proto https
    default_backend web_backends

Certificate changes must be checked for file permissions and handshake before reload. The certificate subject, expiration, and protocol in openssl s_client output should match expectations; example output is marked as such and not fabricated here.

sudo test -r /etc/haproxy/certs/<cert_file>.pem
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
openssl s_client -connect <VIP>:443 -servername <business_domain> -brief </dev/null

Correctly passing client source and protocol information enables backends to generate correct redirects, audit logs, and rate limits. X-Forwarded-For can only be trusted from controlled proxy subnets; backends must not blindly trust headers sent by public clients.

frontend https_frontend
    bind :443 ssl crt /etc/haproxy/certs/<cert_file>.pem
    option forwardfor
    http-request set-header X-Forwarded-Proto https
    default_backend web_backends

HTTP health checks must verify real health paths, expected status codes, and reasonable timeouts. Check paths should avoid database writes and user session dependencies.

backend web_backends
    option httpchk GET /<health_check_URL>
    http-check expect status 200
    default-server inter 2s fall 3 rise 2
    server web1 <backend_addr1>:<port> check
    server web2 <backend_addr2>:<port> check

Use curl to validate health endpoint status, response headers, and response time. HTTP 200 is not a sufficient condition, but abnormal status or slowness should prevent the backend from being marked healthy.

curl -sS -D - -o /dev/null --max-time 3 \
    http://<backend_addr1>:<port>/<health_check_URL>

Cookie persistence can support stateful session migration, but pushing sessions to shared storage or stateless tokens is preferred. Persistence reduces balancing effectiveness, and backend failures still require defined rebalancing behavior.

backend web_backends
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server web1 <backend_addr1>:<port> check cookie web1
    server web2 <backend_addr2>:<port> check cookie web2

Canary releases typically use a header or cookie as an explicit identifier. Do not split traffic by random percentage without recording version tags, otherwise rollback makes it hard to correlate users, requests, and backend versions.

frontend http_frontend
    bind :80
    acl canary hdr(X-Canary) -m str enabled
    use_backend api_canary if canary
    default_backend api_stable

backend api_stable
    server stable1 <backend_addr1>:<port> check

backend api_canary
    server canary1 <backend_addr2>:<port> check

Before canary, validate stable and canary backends with explicit headers. Backends should log version or instance identifiers; seeing the same VIP on the client side does not prove correct traffic splitting.

curl -sS -H 'X-Canary: enabled' -H 'Host: <business_domain>' \
    http://<VIP>/<health_check_URL>
curl -sS -H 'Host: <business_domain>' \
    http://<VIP>/<health_check_URL>

Retries change request semantics. For idempotent requests like GET/HEAD, retries can be cautiously evaluated; for non-idempotent operations like orders, payments, or database writes, business idempotency keys are required — proxy-layer retries alone cannot solve this.

backend api_stable
    option redispatch
    retries 2
    retry-on conn-failure empty-response 500 502 503 504
    server stable1 <backend_addr1>:<port> check

Before enabling retries, use access logs to count methods, status codes, and backend failure types. Total 5xx counts alone cannot distinguish connection failures, timeouts, business errors, and proxy retries.

journalctl -u haproxy --since '30 min ago' --no-pager \
    | rg ' 5[0-9][0-9] |sC|SC|SD|SH' | tail -n 120

4. Validate with Observability and Load Testing, Not Impressions

If the Prometheus exporter is enabled, metric names follow what the exporter actually exposes. Both Layer 4 and Layer 7 should observe sessions, queues, errors, and backend health; Layer 7 should additionally track request status, routing rule hits, and TLS errors.

frontend metrics
    bind 127.0.0.1:8404
    http-request use-service prometheus-exporter if { path /metrics }
    no log

Verify the exporter only listens locally and check its content. Do not expose unauthenticated metrics ports to the public internet.

curl --fail --silent http://127.0.0.1:8404/metrics | head -n 30

The following session metrics are examples based on actual exporter output; observe whether frontend current sessions continuously approach capacity, not just CPU.

sum by (proxy,svname) (
    haproxy_frontend_current_sessions{job="<haproxy_job_name>"}
)

The following backend queue metrics are likewise based on actual exporter output. Queue growth with low CPU may indicate backend connection limits, slow requests, or fewer healthy instances.

max by (proxy,svname) (
    haproxy_backend_current_queue{job="<haproxy_job_name>"}
)

Layer 7 ingress needs alerts on status code ratios. Status codes must be correlated with backend logs: proxy 503, application 500, and upstream timeouts require different remediation.

sum(rate(haproxy_frontend_http_responses_total{
    job="<haproxy_job_name>",code=~"5.."
}[5m]))
/
sum(rate(haproxy_frontend_http_responses_total{
    job="<haproxy_job_name>"
}[5m]))

HTTP load testing should fix concurrency, duration, request body, keepalive, and TLS conditions. Load tests generate real traffic pressure and should only run against approved canary ingresses and read-only paths.

wrk -t4 -c300 -d120s --latency \
    -H 'Host: <business_domain>' \
    https://<VIP>/<health_check_URL>

TCP long-connection scenarios cannot be validated with single-shot HTTP QPS. At minimum, replicate the real model on connection count, hold time, disconnect/reconnect, and backend connection distribution.

for i in $(seq 1 50); do
    nc -z -w 3 <VIP> <frontend_port> &
done
wait
ss -tan state established '( sport = :<frontend_port> )' | wc -l

5. Release, Switch, and Rollback

Before switching, back up configuration, run syntax check, record current backend state and monitoring baseline. Configuration correctness does not guarantee routing correctness; canary ingress should validate paths, headers, TLS, health checks, and source IP before release.

sudo cp -a /etc/haproxy/haproxy.cfg \
    /var/backups/haproxy.cfg.$(date +%Y%m%d_%H%M%S)
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | head -n 30

After single-node canary reload, verify service status, listening ports, and read-only health interface. If currently carrying long connections, also observe whether old connections are retained or gracefully closed as expected.

sudo systemctl reload haproxy
systemctl is-active haproxy
ss -ltnp '( sport = :80 or sport = :443 or sport = :<frontend_port> )'
curl --fail --silent --show-error https://<VIP>/<health_check_URL> >/dev/null

On routing errors, TLS handshake failures, backend mass down, error rate spikes, or tail latency regression, stop rollout expansion and restore the last verified configuration. Run syntax check again before reload to avoid turning rollback into a second incident.

sudo cp -a <backup_dir>/haproxy.cfg /etc/haproxy/haproxy.cfg
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

Post-rollback, verify frontend, backend, and key error logs. If issues persist, root cause may lie in backend, network, DNS, certificates, or clients — not the load balancing mode itself.

systemctl is-active haproxy
echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | head -n 50
journalctl -u haproxy --since '10 min ago' --no-pager | tail -n 160
curl -vkI --max-time 5 https://<VIP>/<health_check_URL>

Final Thoughts

Enterprise selection is often not binary: an outer Layer 4 can handle arbitrary TCP/TLS passthrough and massive connections, while an inner Layer 7 handles HTTP routing, canary, and security governance. The key is not the layer number, but explicitly defining protocol boundaries, TLS termination, client IP preservation, health check depth, timeout/retry semantics, monitoring metrics, and rollback capability in the design — and validating them with real traffic.

Design reviews must also discuss failure domains. Proxy instance counts, availability zone distribution, configuration deployment channels, and certificate dependencies can all become common failure points. More backends do not improve availability if all ingress depends on an unavailable certificate directory, the same DNS resolution, or the same configuration deployment pipeline. Including ingress proxies in capacity drills, certificate renewal drills, and rollback drills is what turns selection into an operable architecture.

The final runbook must clarify who adjusts weights, who maintains certificates, who interprets health check failures, who approves retry policies, and which metrics trigger a release stop. The load balancer easily becomes the "component everyone depends on but no one fully owns"; defining responsibilities, permissions, and observability surfaces is often more important than adding another routing rule.