Why Nginx Returns 400 Bad Request and How to Fix It

Fault Overview

A sudden surge of 400 errors appeared in Nginx error logs after a configuration reload during a sensitive service update.

Incident Details

The engineer rolled back the configuration, then re‑applied a new version and observed many lines like:

1.1.1.1 - - [21/Feb/2017:13:53:00 +0800] "GET /x/get?id=hh2&aid=11642618&..." 400 166 "-" "-" "223.252.221.10" "0.000" "-" "-"

These 400 responses can disrupt downstream business logic.

Investigation Process

Checked the client request method; it used a telnet‑style manual HTTP request without a proper Host header.

Reproduced the issue in a local Nginx test environment using the same request.

Compared old and new configuration files with vimdiff to find differences.

Key Configuration Differences

Old configuration (relevant parts) :

location ~ ^/(api|newapi|admin)/  {
    proxy_set_header X-REAL-IP $remote_addr;
    proxy_pass http://myfcgi-take-proxy;
}

New configuration (relevant parts) :

location ~ ^/(api|newapi|admin)/  {
    set $xheader $remote_addr;
    if ($http_x_forwarded_for != '') {
        set $xheader $http_x_forwarded_for;
    }
    proxy_set_header X-Real-IP $xheader;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass http://myfcgi-take-proxy;
}

Adding proxy_set_header Host $http_host; caused an empty Host header to be forwarded when the client did not supply one.

Root Cause Analysis

According to the HTTP/1.1 RFC, a request without a valid Host header must be answered with 400 Bad Request. Nginx follows this rule, so the empty Host header from the client triggered the error.

The variable $host resolves to the server name when the Host header is missing, while $http_host reflects the raw header value (empty in this case). Replacing $http_host with $host fixed the problem.

Additional Findings

Large client headers can also produce 400 errors; the engineer tested by setting client_header_buffer_size 128k; and large_client_header_buffers 4 128k;, which reproduced the issue.

Health‑check tools (e.g., Nagios check_tcp or keepalived tcp_check) open a TCP connection without sending any data, resulting in an empty Host header and a 400 log entry.

Lessons Learned

Understand Nginx variables and their semantics before modifying proxy headers.

Always test configuration changes in a staging environment and perform a gray‑release or low‑traffic rollout.

Rely on official documentation rather than unverified online snippets.

Be aware that health‑check mechanisms can generate false‑positive 400 logs.

Best Practices for Log Analysis

Use centralized log platforms such as ELK (Elasticsearch, Logstash, Kibana) with appropriate grok patterns to visualize and alert on Nginx error patterns.