Nginx Log Analysis: Quickly Identify Abnormal Requests & Attack Sources

Problem Background

Nginx logs are the first scene for operations and security. When issues arise, teams ask: who is hitting this API? Is it a crawler or a real attack? Is slowness from upstream or client? Are 5xx spikes from backend crashes or Nginx itself?

The author describes two typical tickets:

Business complains of sudden slowness. Access logs show no obvious 5xx, but $upstream_response_time rose from 50ms to 2s. Error logs reveal upstream timed out. Root cause: a PHP-FPM child process hung, causing the whole chain to queue.

Security alerts flood in. WAF catches union select, or 1=1, eval(, base64_decode(. Access log shows a single IP sending 800+ payload requests in one minute. Blocking that IP restored peace.

Both follow the same troubleshooting mainline: understand log format, aggregate anomalies with minimal steps, trace evidence to specific IP/URL/UA/Referer, turn "seeing the problem" into "solving + verifying".

Applicable Scenarios

The article covers: sudden site/API slowdown; multi-GB access logs; error_log spikes ( connect() failed, upstream timed out, no live upstreams, client sent invalid header line); WAF SQL injection alerts; CC attacks; scanner probes ( /admin, /.env, /wp-login.php, /phpmyadmin); abnormal 404/499/502/503/504 aggregation; high-frequency IP blocking; hotlink protection; log disk exhaustion; upstream latency breakdown (connect, header, body); client geography/UA/peak analysis.

Not covered: full ELK/ClickHouse/Loki setup; HTTPS/TLS certificate config; deep Lua/WAF tuning.

Core Knowledge Points

1. access_log Field Composition

Default combined format:

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

. Production recommends custom JSON log_format with fields: time_iso8601, remote_addr, x_forwarded_for, request_method, request_uri, server_name, request_time, upstream_addr, upstream_response_time, upstream_connect_time, upstream_header_time, upstream_status, status, bytes_sent, connection, connection_requests, http_referer, http_user_agent, scheme, gzip_ratio. Each field's meaning and typical use are tabulated.

Note: $request_time is total time from first client byte to last byte sent; $upstream_response_time is time to upstream response end. Comparing them distinguishes client-network slowness from upstream application slowness.

2. error_log Levels & Typical Entries

Levels: debug (requires ngx_debug_module, not for production), info, notice, warn, error, crit, alert, emerg. Common entries: connect() failed (113: No route to host), upstream timed out (110: Connection timed out), no live upstreams, client sent invalid header line, limiting requests, excess: 10 by zone "req_zone", SSL_do_handshake() failed.

3. Log Rotation & Retention

logrotate

default: daily, keep 14 days, compress. Example /etc/logrotate.d/nginx with postrotate sending USR1 signal to Nginx master to reopen logs. Without USR1, Nginx continues writing to old inode, disk not freed.

4. Attack Log Signatures

CC attack: same IP, second-level burst; URL highly concentrated; request_time short but bytes_sent normal; $connection_requests stays 1 (no keepalive).

Scanner: many 404s, long paths with unusual chars; UA like sqlmap/1.5, Nmap Scripting Engine, masscan, empty UA, curl / python-requests; empty/fixed Referer; low per-IP volume but concentrated CIDR blocks.

SQLi/XSS: request line/params contain union select, from information_schema, sleep(, benchmark(, or 1=1, --, /*, ' OR, %27, %20or%20; URIs with %, <script>, onerror=, alert(, document.cookie; status codes not always 5xx, some 200 with anomalous payloads.

5. Attack Source Location & Blocking

Location via logs, blocking via Nginx modules: limit_req_zone + limit_req (IP rate limit), limit_conn_zone + limit_conn (connection limit), geo + map (IP blocklist), allow / deny (basic ACL), return 403 (direct deny). Strategies: single IP high concurrency → limit_req burst+redirect; single scanner UA → map + return 403; whole country → geo + map (legal compliance required).

6. Command-Line Analysis Trio

awk

, grep, sort | uniq -c plus head / tail. Pipeline pattern:

cat access.log | awk '...' | sort | uniq -c | sort -nr | head

. Key points: awk 'condition{action}' condition can be regex; sort -k 1,1 -n numeric sort first column; uniq -c only works after sort; head -n 100 top 100 lines; wc -l line count.

7. Slow Request Judgment & Handling

Two fields: $request_time (client view), $upstream_response_time (upstream view). Relationship: $request_time ≈ $upstream_response_time → upstream slow; $request_time >> $upstream_response_time → Nginx itself (disk buffer, logging, compression, large response); both small but client reports slow → client-to-Nginx network (DNS, TCP, ISP). Precision: $request_time millisecond resolution (1ms min); microsecond needs $msec (1.18+).

8. Reverse Proxy / Proxy Chain Scenarios

To see real client behind SLB/CDN/WAF: set_real_ip_from (trusted proxy IPs), real_ip_header X-Forwarded-For, real_ip_recursive on (multi-level). Otherwise limit_req keys on proxy IP, rendering limits useless.

Overall Troubleshooting Approach

Confirm phenomenon: business feedback, monitoring curves, sudden vs gradual.

Log collection: verify access_log/error_log writing; nginx -V for compile params; nginx -T | grep log_format for active format.

Metric query: error_log error codes & frequency.

Dimension breakdown: TOP by time window / status code / IP / URL / UA.

Correlation: map anomalous IP/URL/UA to business hours, orders, attack logs.

Remediation: rate limit, block, reroute, WAF, upstream tuning.

Verification: re-test or observe, confirm root cause eliminated.

Rollback: keep previous config for instant revert on false positives.

Each step: observe first, then act; avoid direct restart or config change.

Practical Steps (12 Steps)

Step 1: Confirm Log Path, Format, Rotation

Commands: nginx -V 2>&1 | grep -i 'log\|pid',

nginx -T 2>/dev/null | grep -E '^(log_format|access_log|error_log)'

, ls -l /var/log/nginx/, cat /etc/logrotate.d/nginx. Expect JSON log_format with escape=json, proper paths, rotated .gz files, postrotate with kill -USR1. Anomalies: config syntax errors, missing rotated logs (check logrotate -d), USR1 not reaching master (PID path).

Step 2: Custom log_format for Slow Requests & Attack Features

Edit /etc/nginx/nginx.conf in http {} block. Use JSON format with all key fields. Add buffer=32k flush=5s to reduce I/O (max 5s log loss risk). Reload: nginx -t && nginx -s reload. Verify latest log line ends with JSON brace. Risk: buffer/flush may lose logs on crash/power loss; for payment/order systems set flush=1s or disable buffer.

Step 3: Top N Client IPs

For combined format:

awk -F'"' '{print $4}' access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 20

. For JSON: use jq or Python to extract remote_addr. Exclude health-check IPs (127.0.0.1, SLB CIDR, monitor IPs). Single IP >30% and not search engine/monitor → likely attack; low share but concentrated URL → crawler.

Step 4: Top N URLs

Combined:

awk '{print $7}' access.log | sort | uniq -c | sort -nr | head -n 20

. JSON:

jq -r '.request_uri' access.log | sort | uniq -c | sort -nr | head -n 20

. Concentrated /api/list → possible刷接口 or stress test; /.env, /wp-login.php → scanner; /favicon.ico high → missing icon.

Step 5: 5xx Spike by Minute

awk 'BEGIN{IGNORECASE=1} /"status":[5-9][0-9][0-9]| [5-9][0-9][0-9] / { match($0, /\[([0-9]{2}\/[A-Za-z]+\/[0-9]{4}):([0-9]{2}):([0-9]{2}):/, m); minute=m[1]":"m[2]; count[minute]++ } END {for (k in count) print count[k], k}' access.log | sort -nr | head -n 20

. Concentrated minute → correlate with deploy/cron; slow rise → backend leak/resource exhaustion.

Step 6: Slow Requests (request_time > 1s)

Extract IP, URL, latency. For JSON:

grep -E 'request_time":(1[0-9]|[2-9][0-9]|[0-9]{3,})' access.log | awk -F'"' '{for(i=1;i<=NF;i++){if($i~/request_time/){rt=$(i+2)}; if($i~/remote_addr/){ip=$(i+2)}; if($i~/request_uri/){uri=$(i+2)}}} rt+0>=1{print rt, ip, uri}' | sort -nr | head -n 50

. Same IP all slow → client network; same URL all slow → backend/DB; same $upstream_addr all slow → single backend fault.

Step 7: Upstream vs Nginx Slowness

Compare $request_time and $upstream_response_time. rt ≈ urt both large → backend slow; rt >> urt → Nginx (logging/buffer/compression/slow client); urt=- and st=502 → Nginx couldn't connect upstream.

Step 8: error_log Error Classification

grep -oE '\[(error|warn|crit)\] [0-9+#, ]+\*[0-9]+: .*' error.log | grep -oE '(no live upstreams|connect\(\) failed|upstream timed out|SSL_.*|client sent invalid|limiting requests|.*directory.*access denied)' | sort | uniq -c | sort -nr | head -n 20

. upstream timed out → tune proxy_*_timeout /check backend; no live upstreams → upstreams dead, health check missing/failed; limiting requests → verify if legitimate traffic or attack.

Step 9: Identify Attack Types

CC:

awk '{c[$1]++} END {for (k in c) if (c[k] >= 1000) print c[k], k}' access.log | sort -nr | head

(JSON:

jq -r '.remote_addr' | sort | uniq -c | sort -nr | awk '$1 >= 1000' | head

). Scanner:

grep -E '/(admin|wp-login|phpmyadmin|\.env|\.git|backup|\.sql|\.bak)/?(\$|\?|#|)' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr | head

. SQLi:

grep -Ei 'union(\s|%20)+select|select(\s|%20)+\*|or(\s|%20)+1(=|%3d)1|benchmark\(|sleep\(|information_schema' access.log | awk -F'"' '{print $1}' | sort | uniq -c | sort -nr | head

.

Step 10: Mitigation with limit_req / map / geo

Create /etc/nginx/conf.d/security.conf:

limit_req_zone $binary_remote_addr zone=req_zone:10m rate=30r/s;

;

map $http_user_agent $bad_ua { default 0; "~*sqlmap" 1; "~*nmap" 1; "~*masscan" 1; "~*(curl|wget|python-requests)" 1; "~^$" 1; }

; geo $geo_country_code $is_high_risk { default 0; XX 1; }. In server: limit_req zone=req_zone burst=60 nodelay;; if ($bad_ua) { return 403; }; if ($is_high_risk) { return 403; }. Include proxy_params for headers. Reload and verify: tail -f access.log | grep -E ' 403 '. Watch for false positives (legitimate users get 403) and limit key mismatch in proxy chains (must use $binary_remote_addr with real_ip).

Step 11: Verification Loop

Check attack IP status codes:

grep '203.0.113.45' access.log | tail -n 50 | awk '{print $9}' | sort | uniq -c

→ expect 50 403. Test normal UA (should not be blocked), sqlmap UA (should get 403).

Step 12: Log Rotation, Disk Usage, Cleanup

Use logrotate -d (dry-run) then logrotate -f (force). Monitor df -h /var/log. Delete old: find /var/log/nginx -name '*.gz' -mtime +30 -ls then -delete. Risks: rm access.log + reload doesn't free disk (fd holds old inode); use > access.log truncate then kill -USR1. Don't echo "" > access.log under load. Batch delete with -mtime +30.

Common Commands Cheat Sheet

Table of commands with purpose and key judgment (converted to list): nginx -V – view compile params (debug, modules) – check module presence. nginx -t – test config syntax – run after every change. nginx -T – dump full config – troubleshoot default differences. nginx -s reload – graceful reload – never use restart. tail -F /var/log/nginx/access.log – real-time follow – -F more reliable than -f.

awk '{print $7}' access.log | sort | uniq -c | sort -nr | head

– TOP N URL – field position per combined format. grep -E ... | wc -l – count matching requests – combine with cut for time windows. grep 'IP' access.log | awk '{print $9}' | sort | uniq -c – single IP status distribution – attack vs normal vs rate-limited. cat access.log | python3 -c '...json...' – parse JSON logs – works without jq. jq -r '.remote_addr' access.log – extract JSON field – install jq. grep -E ' 5[0-9][0-9] ' – filter 5xx – note double-space wrapping. iptables -L -n -v | grep DROP – firewall drops – confirm block source. wc -l access.log – line count – rough file size gauge.

Configuration Examples

1. Nginx http Block (with log_format)

Full nginx.conf with JSON log_format main escape=json, access_log ... buffer=32k flush=5s, limit_req_zone, limit_conn_zone, map $http_user_agent $bad_ua.

2. Server Config (Rate Limit & Blocklist)

Upstream with max_fails=3 fail_timeout=10s, keepalive 64. HTTP to HTTPS redirect. SSL config. if ($bad_ua) { return 403; }. Global limit_req zone=req_zone burst=60 nodelay;, limit_conn conn_zone 50;. Proxy settings with timeouts and buffers. Static location with expires 7d, access_log off.

3. logrotate

Standard daily, rotate 14, compress, delaycompress, sharedscripts, postrotate with kill -USR1 $(cat /var/run/nginx.pid).

4. iptables Fallback (Extreme Cases)

iptables -I INPUT -s 203.0.113.45 -j DROP

. Warning: if behind SLB/CDN, blocking SLB CIDR breaks own access; prefer Nginx deny or block at WAF/CDN layer.

Log & Metric Observation Methods

1. stub_status for Concurrency & Connections

Enable on localhost: stub_status on; allow 127.0.0.1; deny all;. curl -s http://127.0.0.1:8080/nginx_status. Output: Active connections, accepts/handled/requests, Reading/Writing/Waiting. Judgment: Active connections rising → connection pileup; Waiting high → slow connections eating workers; accepts != handled → kernel drops, tune worker_rlimit_nofile / somaxconn.

2. vts Module (Optional)

Requires recompile with --add-module=.../nginx-module-vts. Not detailed.

3. Log Volume Monitoring

Weekly: ls -lhS /var/log/nginx/, du -sh /var/log/nginx. Anomaly: single day > tens of GB → verbose format, attack, or no rotation; single file > 2GB → analysis slows.

4. Key Metric Alert Thresholds (Examples)

5xx ratio: 5xx_total / total_requests >1% warn, >5% critical.

P99 latency: request_time 99th percentile >1s watch, >3s alert.

Wait connections: Waiting > worker_connections * 0.8.

Disk log usage: du -sh /var/log/nginx >80% alert.

All thresholds must be tuned to business baselines; separate peak/off-peak thresholds.

Troubleshooting Paths (Decision Trees)

Path 1: Site Slowness

Phenomenon: business reports seconds to load, RT up.

Check error_log for upstream timed out, connect() failed, no live upstreams → if hit, go upstream branch.

Compare $request_time vs $upstream_response_time: close & large → backend slow; request_time >> upstream_response_time → Nginx (logging/compression/buffer).

Check stub_status: high Waiting → slow connections.

Client slow → analyze top client IP, UA, geo.

Path 2: 5xx Surge

Alert on 5xx.

Minute-level 5xx aggregation (Step 5).

Bucket by 502/503/504/500/499: 502 → upstream unreachable; 503 → max_fails triggered upstream circuit break; 504 → upstream timed out; 500 → backend error propagated; 499 → client closed (slow or rate-limited).

Correlate error codes with error_log & backend logs at same timestamps.

Path 3: CC / Scanner Attack

Symptoms: connection/bandwidth saturation, specific URL high freq.

Top IP (Step 3).

URL concentration (Step 4).

UA tool signatures (Step 9).

Decision: single IP → limit_req_zone or iptables; UA cluster → map $http_user_agent + 403; path cluster → location ~* URL + limit_req.

Path 4: Disk Full by Logs

Alert: disk full, access_log fills /var/log. du -sh /var/log/nginx.

Check logrotate: logrotate -d /etc/logrotate.d/nginx, cat /var/lib/logrotate.status.

Remediation: temporary delete old .gz (>3 days) after -ls confirm; long-term: rotate 7, compress, notifempty.

Path 5: Proxy IP Seen Instead of Real Client

Symptom: rate limiting ineffective, all requests appear from single IP.

Verify real_ip_header and set_real_ip_from.

Confirm XFF header not stripped by middle layer.

Ensure limit_req_zone key $binary_remote_addr works with real_ip effective.

Path 6: Legitimate Users Get 429

Reports: normal users cannot access.

Check access_log for concentrated IPs.

Lower threshold or add URI whitelist exceptions.

Adjust client retry strategy (Retry-After header).

Path 7: Persistent Scanner Harassment

High 404/403 sustained.

Add scan patterns to map (e.g., /wp-login.php) → return 444 (close connection without response).

Risk Warnings

rm deletes log: rm access.log && nginx -s reload doesn't free disk (fd holds old inode). Use > access.log truncate then kill -USR1.

truncate / > file: master fd still points to old inode; space freed only after worker releases. Use kill -USR1 $(cat /var/run/nginx.pid) to reopen.

find -delete mass cleanup: wrong condition wipes all production logs. First find ... -ls, then find ... -print | xargs -I{} echo {} simulate, finally -delete.

Rate limit false positives: rate=1r/s burst=0 kills large downloads/sync. Exclude static resources ( GET /static/*) with higher limits or no limit.

Block CIDR mistakes: blocking SLB CIDR cuts off Nginx from upstream. Confirm internal IP ranges & whitelist first.

Reload without test: nginx -s reload without nginx -t leads to parse failure on next start. Always test first.

Log format corruption: custom log_format quote/field errors break downstream ELK. Use escape=json and test with one request before rollout.

Aggressive buffer/flush loses logs: buffer=64k flush=60s loses 60s on crash. Payment/order: flush=1s; normal: flush=5-10s.

Verification Methods

Attack source blocked:

grep 'IP' access.log | tail -n 100 | awk '{print $9}' | sort | uniq -c

→ expect 100 403.

Slow requests back to baseline:

jq -r '.request_time' access.log | awk '{if($1+0 > 1) c++} END{print "slow:", c+0}'

compare pre/post 99th percentile.

Rate limit effective:

for i in {1..50}; do curl -s -o /dev/null -w "%{http_code}
" http://example.com/api/list; done | sort | uniq -c

→ mix of 200 and 429.

Reload success:

nginx -t && nginx -s reload && systemctl status nginx && ps aux | grep nginx | grep -v grep

→ master PID unchanged, worker PIDs changed.

logrotate success:

logrotate -d /etc/logrotate.d/nginx && logrotate -f /etc/logrotate.d/nginx && ls -lh /var/log/nginx/

→ old .1 / .gz appear, new file has fresh lines.

False positive check:

grep '403' access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head

→ if top 403 IPs include internal/health-check/whitelisted UA, those are false blocks.

stub_status recovery: watch -n1 'curl -s http://127.0.0.1:8080/nginx_status' → Active connections drop to normal baseline after mitigation.

Rollback Plans

Config change: backup cp nginx.conf nginx.conf.bak$(date +%s); rollback

cp nginx.conf.bakTIMESTAMP nginx.conf && nginx -t && nginx -s reload

.

limit_req / map adjust: backup current map & zone; rollback with backup files.

logrotate not rotating: debug logrotate -d; restore postrotate (USR1).

iptables mis-block internal: iptables -L -n -v | grep DROP; delete rule iptables -D INPUT -s 203.0.113.45 -j DROP.

Custom log_format breaks downstream: switch access_log back to combined; test new format in staging first.

Log cleanup accidental delete: restore from offsite backup/archive.

Production Considerations

Don't truncate logs directly: echo "" > access.log leaves fd on old inode, disk not freed, may break ELK JSON parsing. Use

echo -n "" > access.log; kill -USR1 $(cat /var/run/nginx.pid)

.

Log sampling & split by business: multiple access_log directives per server_name/location for domain-level troubleshooting.

Real client IP behind proxy: must set set_real_ip_from (SLB/CDN CIDRs), real_ip_header X-Forwarded-For, real_ip_recursive on.

High concurrency buffer: buffer=32k flush=5s reduces I/O jitter but risks 5s loss. Payment/order: buffer=8k flush=1s or no buffer.

Rate limit thresholds per business: large uploads need lower rate; static resources wider; login/payment/verification stricter.

Log alerting & archiving: weekly offsite archive (OSS/S3); retain 7-14 days locally; monitor /var/log >70% alert.

Process & file descriptors: worker_rlimit_nofile match ulimit -n; tune somaxconn and tcp_max_syn_backlog.

Security hardening: don't expose stub_status publicly; avoid /tmp for logs; encrypt log transit.

Multiple Nginx instances: each instance's PID path must match its logrotate postrotate.

Don't restart on anomalies: nginx -s reload suffices for most config errors; systemctl restart nginx drops long connections, only for worker deadlock.

Summary

Nginx log analysis is not trickery; it's a closed loop: log full fields (custom JSON log_format with $request_time, $upstream_response_time, $upstream_addr), aggregate anomalies (by IP, URL, UA, time window), evidence chain to specifics (map anomaly to exact machine, endpoint, client), stable mitigation ( limit_req_zone + map, rules rollbackable, canary, verifiable), log rotation ( logrotate + USR1 + disk alerts), verification loop (post-mitigation monitor status codes, false positives, baseline recovery). With this loop, every production issue can be located in minutes; security alerts are triaged via CC/scanner/SQLi buckets, mitigated, then blocked. Troubleshooting proficiency is muscle memory of "log-command-config" triad. Memorize core commands, field meanings, config knobs; daily review alerts, write bad cases to wiki; next time same pattern appears, two steps to resolution.