How to Quickly Spot Anomalous Requests and Attack Sources Using Nginx Logs

Problem Background

Nginx logs are the first source for operations and security when a service experiences slowdown, high error rates, or attacks. Typical symptoms include sudden latency, large access_log files, error messages such as upstream timed out, and alerts for SQL injection or CC attacks.

Core Knowledge

1. Access Log Fields (JSON format)

log_format main escape=json '{
    "time":"$time_iso8601",
    "remote_addr":"$remote_addr",
    "x_forwarded_for":"$http_x_forwarded_for",
    "request_method":"$request_method",
    "request_uri":"$request_uri",
    "server_name":"$server_name",
    "request_time":"$request_time",
    "upstream_addr":"$upstream_addr",
    "upstream_response_time":"$upstream_response_time",
    "upstream_connect_time":"$upstream_connect_time",
    "upstream_status":"$upstream_status",
    "status":"$status",
    "bytes_sent":"$body_bytes_sent",
    "connection_requests":"$connection_requests",
    "http_user_agent":"$http_user_agent",
    "http_referer":"$http_referer"
}';
access_log /var/log/nginx/access.log main buffer=32k flush=5s;
error_log  /var/log/nginx/error.log warn;

Key fields: $remote_addr – client IP (direct) $http_x_forwarded_for – XFF header for real client behind proxies $request_time – total time from first byte received to last byte sent (client view) $upstream_response_time – time spent waiting for upstream response $status – HTTP status code $http_user_agent – User‑Agent string $http_referer – Referer header

2. Error Log Levels

debug

– detailed debugging (requires ngx_debug_module) info – informational messages notice – notable events warn – warnings error – errors affecting responses crit – critical errors alert / emerg – system‑level failures

Common production messages:

connect() failed (113: No route to host) while connecting to upstream
upstream timed out (110: Connection timed out) while reading response header from upstream
no live upstreams while connecting to upstream
client sent invalid header line
limiting requests, excess: 10 by zone "req_zone"
SSL_do_handshake() failed

3. Log Rotation

Typical logrotate configuration (daily, keep 14 compressed backups):

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

The USR1 signal tells Nginx to reopen log files; otherwise the old inode remains open and disk space is not freed.

4. Attack Signatures

CC (Challenge Collapsar) attack : single IP > 1000 requests per second, short $request_time but normal $bytes_sent, $connection_requests stays at 1 (no keep‑alive).

Scanner : many 404s, long or unusual URLs, User‑Agents like sqlmap, nmap, masscan, empty UA, curl, wget.

SQL injection / XSS : request line contains keywords such as union, select, or 1=1, benchmark, sleep, etc.; often accompanied by encoded characters ( %, <script>).

Step‑by‑Step Investigation Process

Confirm phenomenon : collect business feedback and monitoring graphs.

Check log collection : verify access_log and error_log paths, formats, and modules using nginx -V and nginx -T.

Aggregate metrics : use awk, grep, sort, uniq -c to find top IPs, URLs, status codes, and time‑window spikes.

Identify slow requests : filter $request_time > 1s and extract IP, URL, duration.

Determine bottleneck :

If $request_time ≈ $upstream_response_time → upstream is slow.

If $request_time >> $upstream_response_time → Nginx overhead (logging, buffering, large responses).

If both are small but client feels slow → network issue between client and Nginx.

Correlate with error_log : group error messages by cause (e.g., upstream timed out, no live upstreams, limiting requests).

Apply mitigation :

Rate limit per IP:

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s;

and limit_req.

Block malicious User‑Agents with a map that returns 403.

Geo‑block high‑risk countries using geo + map.

Use iptables for L4 blocking as a last resort.

Validate mitigation : re‑run aggregation commands, ensure offending IP now receives 403/429, and confirm normal traffic returns to baseline.

Rollback plan : keep original configuration backups, test with nginx -t before reload, and restore backups if needed.

Common Command Reference

nginx -V

– show compile options and loaded modules. nginx -t – test configuration syntax (must succeed before reload). nginx -T – print full configuration (useful to compare defaults). nginx -s reload – graceful reload without dropping connections. tail -F /var/log/nginx/access.log – follow log across rotations.

awk '{print $7}' access.log | sort | uniq -c | sort -nr | head

– top N URLs (for combined format).

jq -r '.remote_addr' access.log | sort | uniq -c | sort -nr | head

– top N client IPs (for JSON logs).

grep -E 'union|select|or 1=1|benchmark|sleep' access.log | wc -l

– count possible SQL‑injection attempts. iptables -L -n -v | grep DROP – view firewall drop rules. df -h /var/log – monitor disk usage of log directory.

Configuration Examples

1. Global HTTP Block with JSON Log Format

user  nginx;
worker_processes auto;
worker_rlimit_nofile 65535;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 4096;
    multi_accept on;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    sendfile      on;
    tcp_nopush    on;
    tcp_nodelay   on;
    keepalive_timeout 65;
    keepalive_requests 1000;

    # JSON log format
    log_format main escape=json '{
        "time":"$time_iso8601",
        "remote_addr":"$remote_addr",
        "x_forwarded_for":"$http_x_forwarded_for",
        "request_method":"$request_method",
        "request_uri":"$request_uri",
        "server_name":"$server_name",
        "request_time":"$request_time",
        "upstream_addr":"$upstream_addr",
        "upstream_response_time":"$upstream_response_time",
        "upstream_status":"$upstream_status",
        "status":"$status",
        "bytes_sent":"$body_bytes_sent",
        "connection_requests":"$connection_requests",
        "http_user_agent":"$http_user_agent",
        "http_referer":"$http_referer"
    }';

    access_log /var/log/nginx/access.log main buffer=32k flush=5s;
    error_log  /var/log/nginx/error.log warn;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s;
    limit_req_status 429;

    # UA blacklist
    map $http_user_agent $bad_ua {
        default 0;
        "~*sqlmap"      1;
        "~*nmap"        1;
        "~*masscan"     1;
        "~*python-requests" 1;
        "~*curl"        1;
        "~*wget"        1;
        "~^$"           1;   # empty UA
    }

    include /etc/nginx/conf.d/*.conf;
}

2. Server Block with Rate Limiting and UA Blacklist

upstream backend {
    server 10.0.0.7:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.8:8080 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate     /etc/nginx/ssl/example.crt;
    ssl_certificate_key /etc/nginx/ssl/example.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # Block bad UA
    if ($bad_ua) { return 403; }

    # Global rate limiting
    limit_req zone=req_zone burst=60 nodelay;

    access_log /var/log/nginx/example.access.log main buffer=32k flush=5s;
    error_log  /var/log/nginx/example.error.log warn;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 3s;
        proxy_send_timeout    30s;
        proxy_read_timeout    30s;
        proxy_buffer_size      16k;
        proxy_buffers          8 16k;
        proxy_busy_buffers_size 32k;
    }

    location ~* \.(js|css|jpg|jpeg|png|gif|ico|svg|woff2?)$ {
        expires 7d;
        add_header Cache-Control "public, max-age=604800, immutable";
        access_log off;
    }
}

3. Logrotate Configuration

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

4. Optional iptables Block (L4 fallback)

iptables -I INPUT -s 203.0.113.45 -j DROP
iptables -L -n -v | grep DROP

Note: In a reverse‑proxy scenario, blocking the upstream SLB IP range with iptables can cut off all traffic. Prefer Nginx deny rules or block at the CDN/WAF layer.

Risk Reminders

Using buffer=32k flush=5s may lose up to 5 seconds of logs on power loss. For payment or order systems set flush=1s or disable buffering.

Never delete log files with rm without sending USR1; the inode stays open and disk space is not reclaimed.

Truncating with echo "" > file can split logs under heavy load; use > file followed by kill -USR1 instead.

When cleaning old logs, always preview with -ls before using -delete.

Rate‑limit keys must be the real client IP. In a reverse‑proxy setup configure set_real_ip_from and real_ip_header X-Forwarded-For so that $binary_remote_addr reflects the true client.

Verification Checklist

Confirm blocked IP now returns 403:

grep '203.0.113.45' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Check slow‑request count after mitigation:

jq -r '.request_time' /var/log/nginx/access.log | awk '{if($1+0>1) c++} END{print "slow:",c}'

Validate rate‑limit effect:

for i in {1..50}; do curl -s -o /dev/null -w "%{http_code}
" http://example.com/api/list; done | sort | uniq -c

Ensure Nginx reload succeeded: nginx -t && nginx -s reload && systemctl status nginx Verify logrotate created a new .1.gz file and the old file is no longer growing.

Search for unexpected 403 or 429 responses to detect false positives.

Rollback Procedures

Before any change, copy the original file:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak$(date +%s)

After a failed reload, restore the backup and reload again:

cp /etc/nginx/nginx.conf.bakTIMESTAMP /etc/nginx/nginx.conf && nginx -t && nginx -s reload

For limit_req or map adjustments, keep a copy of the snippet and revert if the block list causes legitimate traffic to be denied.

If logrotate stops rotating, run a dry‑run to debug and ensure the

postrotate

kill -USR1

command points to the correct PID file.

To undo an iptables block, delete the rule: iptables -D INPUT -s 203.0.113.45 -j DROP If a custom log_format breaks downstream parsers, switch back to the standard combined format: access_log /var/log/nginx/access.log combined; When accidental log deletion occurs, restore from the most recent backup or archive (OSS/S3).

Production Tips

Never expose stub_status to the public internet; bind it to 127.0.0.1 only.

Separate static‑asset logs (or disable logging for them) to reduce log volume.

Adjust buffer / flush based on business criticality: flush=1s for financial services, flush=5s for general traffic.

Set worker_rlimit_nofile and system somaxconn high enough to avoid “accept() failed” errors under load.

Monitor stub_status metrics (active connections, waiting, reading, writing) to spot connection‑pool exhaustion.

Define alert thresholds (e.g., 5xx > 1 % of requests, 99th‑percentile request_time > 1 s, disk usage > 80 %).

Keep at least 7 days of raw logs locally; archive older logs to remote storage.

Conclusion

Effective Nginx troubleshooting relies on a complete JSON log format, systematic aggregation of IP/URL/UA/time windows, precise bottleneck identification using $request_time vs $upstream_response_time, and disciplined mitigation via limit_req, map, geo, or firewall rules. Coupled with reliable log rotation, continuous verification, and clear rollback plans, this workflow enables minute‑level detection and remediation of performance degradations and security attacks while preserving log integrity for downstream analysis.