Why a Simple Nginx Change Made All Gateway Requests Return 400 (And How to Fix It)

In a project that routes public traffic through Spring Cloud Gateway, two Nginx reverse proxies sit in front of the gateway. The network path is

domain → Nginx → F5 load balancer → gateway → backend services

.

During a production change, the Nginx team swapped the two proxy machines. Immediately after traffic was switched to the new Nginx instances, all gateway API calls started returning HTTP 400. Rolling back to the old Nginx restored normal responses, and the incident was classified as an SRE P1.

Initial log inspection showed that OPTIONS requests returned 204, while GET/POST requests consistently returned 400, and the gateway itself logged no errors during the failure window.

Comparing the old and new Nginx configurations revealed two key differences in the new setup:

Definition of an upstream http_gateways block with keepalive 30.

Enabling HTTP/1.1 with proxy_http_version 1.1; and clearing the Connection header via proxy_set_header Connection "";.

Reproducing the issue in a test environment using

curl -v -X GET http://10.100.8.11:9104/wechat-web/actuator/info

returned 400. Removing the two lines proxy_http_version 1.1; and proxy_set_header Connection ""; made the request succeed with 200.

A deeper investigation with

tcpdump -vv -i ens192 host 10.100.22.48 and tcp port 8081 -w /tmp/ng400.cap

captured the failing requests. The captured HTTP headers showed a Host: http_gateways value. According to the HTTP/1.1 specification, the Host header must be a valid hostname; underscores are not allowed. Nginx’s default proxy_set_header Host $proxy_host; uses the upstream name as the Host value, so the underscore in http_gateways produced an illegal Host header, causing the gateway to reject the request with 400.

Spring Cloud Gateway’s request handling code confirms this behavior. In ReactorHttpHandlerAdapter.apply(...), a URISyntaxException triggers reactorResponse.status(HttpResponseStatus.BAD_REQUEST);, which is logged only at DEBUG level, explaining the absence of error logs in production.

To resolve the problem, the Nginx configuration was updated to explicitly set a valid Host header: proxy_set_header Host $host; Alternatively, renaming the upstream to remove the underscore (e.g., http-gateways) or hard‑coding the domain in proxy_set_header Host "example.com"; also works. After adding the explicit Host header, the test requests returned 200, confirming the fix.

The final working configuration looks like this:

upstream http_gateways {
  server 10.100.22.48:8081;
  keepalive 30;
}
server {
  listen 9104 backlog=512;
  server_name wmg.test.com;
  add_header X-Frame-Options "SAMEORIGIN";
  add_header X-Content-Type-Options "nosniff";
  add_header Content-Security-Policy "frame-ancestors 'self'";
  location / {
    proxy_set_header Host $host;
    proxy_hide_header host;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    client_max_body_size 100m;
    add_header 'Access-Control-Allow-Origin' "$http_origin" always;
    add_header 'Access-Control-Allow-Credentials' 'true' always;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, DELETE, PUT';
    add_header 'Access-Control-Allow-Headers' '...';
    if ($request_method = 'OPTIONS') { return 204; }
    proxy_pass http://http_gateways;
  }
}

The incident demonstrates that seemingly minor configuration differences—such as an upstream name containing an underscore—can break HTTP compliance and cause production outages. Thorough testing of configuration changes and awareness of protocol requirements are essential to avoid repeat incidents.