Quick Nginx Log Analysis Techniques to Spot Abnormal Requests and Attack Sources

Problem background

Nginx logs are the first line of information for operations and security teams. When a service slows down or an attack occurs, stakeholders ask questions like "Who is hitting this endpoint? Is it a crawler or a real attack? Is the latency on the client side, Nginx, or the upstream?"

Typical incidents

Website suddenly slows; $upstream_response_time jumps from 50 ms to 2 s while a PHP‑FPM worker hangs.

WAF alerts show payloads such as union select, or 1=1, eval( etc., indicating SQL‑injection attempts.

Analysis workflow

Confirm log format and rotation settings using nginx -V, nginx -T, and cat /etc/logrotate.d/nginx.

Collect key fields via a custom JSON log_format that includes time, client IP, X‑Forwarded‑For, request method, URI, status, request/response times, upstream address, user‑agent and referer.

Use awk, grep, sort, uniq -c and jq to extract top IPs, URLs, 5xx counts, slow requests (e.g., request_time >= 1), and attack signatures.

Compare $request_time (client‑side latency) with $upstream_response_time to decide whether the bottleneck is in Nginx, the network, or the backend.

Classify error log entries by level (debug, info, notice, warn, error, crit, alert, emerg) and by typical messages such as connect() failed, upstream timed out, no live upstreams, client sent invalid header line, limiting requests.

Key command examples

# Verify Nginx compilation flags and active log format
nginx -V
nginx -T | grep -E '^(log_format|access_log|error_log)'

# Top 20 client IPs (JSON logs)
cat /var/log/nginx/access.log | jq -r '.remote_addr' | sort | uniq -c | sort -nr | head -n 20

# Top 20 URLs (combined format)
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

# 5xx per minute aggregation
awk 'BEGIN{IGNORECASE=1} /"status":[5-9][0-9][0-9]/ {match($0, /\[([0-9]{2}\/[A-Za-z]+\/[0-9]{4}):([0-9]{2}):/, m); minute=m[1]":"m[2]; count[minute]++} END{for(k in count) print count[k], k}' /var/log/nginx/access.log | sort -nr | head

# Slow request extraction (>=1 s)
awk '{match($0, /([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/, ip); match($0, /"([A-Z]+) ([^ ]+) HTTP/, req); match($0, /request_time\":([0-9.]+)/, rt) if(rt[1]+0>=1) print rt[1], ip[1], req[2]}' /var/log/nginx/access.log | sort -nr | head

Attack detection patterns

CC (Challenge Collapsar) : single IP > 1000 requests per minute, low $request_time and $bytes_sent. Mitigate with

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s;

and limit_req zone=req_zone burst=60 nodelay;.

Scanners : many 404s for paths like /admin, /wp-login.php, /phpmyadmin. Detect with

grep -E '/(admin|wp-login|phpmyadmin|\.env|\.git|backup|\.sql|\.bak)/'

and block via

map $http_user_agent $bad_ua { default 0; "~*sqlmap" 1; "~*nmap" 1; }

→ return 403;.

SQL injection : payloads containing union select, or 1=1, benchmark(, sleep(, etc. Detect with grep -Ei 'union\s+select|or\s+1=1|benchmark\(|sleep\(' and block similarly.

Mitigation configuration

# /etc/nginx/nginx.conf (excerpt)
log_format main escape=json '{"time":"$time_iso8601","remote_addr":"$remote_addr","x_forwarded_for":"$http_x_forwarded_for","request_method":"$request_method","request_uri":"$request_uri","server_name":"$server_name","request_time":"$request_time","upstream_addr":"$upstream_addr","upstream_response_time":"$upstream_response_time","upstream_status":"$upstream_status","status":"$status","bytes_sent":"$body_bytes_sent","connection_requests":"$connection_requests","http_user_agent":"$http_user_agent","http_referer":"$http_referer"}';
access_log /var/log/nginx/access.log main buffer=32k flush=5s;

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s;
limit_conn_zone $binary_remote_addr zone=conn_zone:10m;

map $http_user_agent $bad_ua {
    default 0;
    "~*sqlmap" 1;
    "~*nmap" 1;
    "~*masscan" 1;
    "~*curl" 1;
    "~*wget" 1;
    "~^$" 1;
}

# Example server block with rate limiting and UA blacklist
server {
    listen 443 ssl http2;
    server_name example.com;
    ssl_certificate /etc/nginx/ssl/example.crt;
    ssl_certificate_key /etc/nginx/ssl/example.key;

    if ($bad_ua) { return 403; }
    limit_req zone=req_zone burst=60 nodelay;
    limit_conn conn_zone 50;

    location / { proxy_pass http://backend; }
    location ~* \.(js|css|png|jpg|gif|svg|woff2?)$ { expires 7d; access_log off; }
}

Log rotation

/etc/logrotate.d/nginx {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Monitoring and alerts

Use stub_status (e.g., curl -s http://127.0.0.1:8080/nginx_status) to watch active connections, waiting queue, and request rates.

Track 5xx ratio, 99th‑percentile request time, waiting connections, and disk usage of /var/log/nginx with simple awk / grep pipelines or external monitoring tools.

Risk reminders

Never delete a log file without signalling Nginx (use kill -USR1 $(cat /var/run/nginx.pid) or truncate with > file).

Be cautious with buffer=... flush=5s; high‑value services (payments, orders) should use flush=1s or no buffer to avoid data loss.

Validate limit_req keys ( $binary_remote_addr) and ensure real_ip_header X-Forwarded-For is correctly set when behind a proxy.

Test any JSON log_format changes on a staging environment before production rollout.

Verification steps

# Verify blocked IP returns 403
grep '203.0.113.45' /var/log/nginx/access.log | tail -n 20 | awk '{print $9}' | sort | uniq -c

# Check slow‑request count after mitigation
jq -r '.request_time' /var/log/nginx/access.log | awk '{if($1+0>=1) c++} END{print "slow requests:",c}'

# Ensure limit_req works
for i in {1..50}; do curl -s -o /dev/null -w "%{http_code}
" https://example.com/api/list; done | sort | uniq -c

# Reload safely
nginx -t && nginx -s reload

Rollback procedures

Backup nginx.conf before changes (e.g., cp nginx.conf nginx.conf.bak$(date +%s)).

If a change breaks the service, restore the backup and reload:

cp nginx.conf.bak... nginx.conf && nginx -t && nginx -s reload

.

For limit_req or map adjustments, revert to the previous file and reload.

If log rotation stops, run logrotate -f /etc/logrotate.d/nginx and verify the USR1 signal is executed.

Production notes

Configure real_ip_header X-Forwarded-For and set_real_ip_from for all trusted proxy CIDR blocks to ensure accurate client IPs.

Separate access logs per virtual host or service to simplify analysis.

Adjust buffer and flush values per business criticality (payments → flush=1s, others → flush=5s).

Keep worker_rlimit_nofile and sysctl net.core.somaxconn high enough for peak traffic.

Conclusion

By defining a comprehensive JSON log format, leveraging lightweight command‑line pipelines, and applying Nginx built‑in rate‑limiting and access‑control modules, operators can quickly pinpoint slow requests, error spikes, and malicious traffic, mitigate them with deterministic rules, and verify the remediation in minutes, all while maintaining safe log rotation and avoiding data loss.