Nginx High Connections: Normal Traffic or Slowloris Attack? Diagnostic Guide
This guide provides a systematic forensic process to distinguish between normal high Nginx connections and slow connection attacks (Slowloris), using Linux commands, Nginx stub_status, connection state analysis, rate comparisons, packet capture, and evidence-based decision matrices for accurate diagnosis and mitigation.
1. Problem Background
Operations staff observed thousands of established connections via ss -s while CPU remained under 10% and bandwidth low. The team debated whether this was a Slowloris attack or normal traffic surge, with conflicting suggestions to increase worker_connections, block IPs, or restart Nginx. The core challenge is establishing a forensic process to differentiate "normal high connections" from "slow connection attacks" before taking action.
2. Applicable Scenarios
Suitable when:
Connection count significantly exceeds baseline via ss / netstat and needs classification.
Site shows "connects but won't load" symptoms with low server utilization.
Monitoring alerts on active connections persist.
Planning rate limiting or hardening and need to understand restrictions and risks.
Direct connections reach Nginx bypassing CDN/WAF.
Not suitable when: CPU at 100% (different troubleshooting path), massive SYN_RECV (SYN Flood), Layer 4 load balancer in front (must identify correct layer), or massive short-connection workloads where high connections are normal.
3. Core Knowledge
3.1 Nginx Connection States
Nginx stub_status divides connections into three categories: reading: Reading client request headers/body. writing: Writing response to client. waiting: Keep-alive idle connections awaiting next request.
Sum equals active connections. Slowloris attacks pile up in reading; normal high concurrency grows waiting.
3.2 TCP State Machine Quick Reference
Key states via ss: ESTABLISHED: Both normal and attack connections appear here; must inspect internal activity. SYN_RECV: High counts indicate SYN Flood or small backlog. TIME_WAIT: Normal in short-connection scenarios. CLOSE_WAIT: Often indicates upstream closed but Nginx hasn't.
Critical: ESTABLISHED ≠ "processing request". Keep-alive, stalled, and attack connections all show as ESTABLISHED.
3.3 Slowloris Variants
Slowloris (slow headers): Sends header lines one by one with long delays.
Slow Body / RUDY: Completes headers but sends body extremely slowly.
Slow Read: Sends request but reads response very slowly.
Common trait: Nginx request processing rate (QPS) very low while connection occupancy rate very high. Normal traffic shows opposite pattern.
3.4 Nginx Event Model & Connection Limits
Nginx uses epoll. Total connection capacity ≈ worker_processes × worker_connections. Three-tier limit hierarchy:
Nginx per-worker limit ( worker_connections)
Process file descriptor limit ( ulimit -n / worker_rlimit_nofile)
Kernel global limit ( fs.file-max / fs.nr_open)
All three must align; raising only worker_connections without adjusting others is ineffective.
3.5 Why "High Connections, Low CPU, Low Bandwidth" = Slowloris
Comparison of metrics:
Metric Normal High Traffic Slowloris Attack
Established connections High, correlates with request rate High or slowly climbing, decoupled from request rate
CPU Fluctuates with load Persistently low (workers waiting on read)
Bandwidth Rises with connections Minimal
Access log Lines continuously generated Attack connections produce no log entries (requests incomplete)
stub_status distribution waiting/reading follow business rhythm reading persistently high, waiting minimal
Per-IP connections Natural distribution (single digits to tens) Many IPs with dozens/hundreds each
New connection rate Matches request arrival rate Far exceeds request completion rateAt least three matching evidence lines required for conclusion.
3.6 Process Attribution
System-wide ss -s includes non-Nginx connections. Must attribute established connections to processes (requires root):
ss -tanp | awk 'NR>1 && /ESTAB/ { n=split($NF,a,"="); gsub(/[")]/,"",a[2]); proc=(a[2]?a[2]:"unknown"); cnt[proc]++ } END { for (p in cnt) print cnt[p], p }' | sort -rn | headExpected output shows nginx dominating. Also separate inbound (port 80/443) vs outbound (upstream) connections.
4. Overall Judgment Framework
Seven-step evidence chain:
Total count & state distribution
Reading/writing/waiting breakdown
Source IP aggregation & concentration
New connection rate vs request completion rate
Access log growth & error codes
Packet capture sampling
Decision matrix conclusion
Principle: Evidence first, then action; every action must be reversible.
5. Practical Steps (with Commands)
5.1 Total Count & State Distribution
ss -sThen state breakdown:
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn5.2 Nginx stub_status Analysis
Enable stub_status (internal only), then: curl -s http://127.0.0.1/nginx_status Key: Reading high + requests growth low = Slowloris signature. Continuous sampling (10s intervals) reveals trends.
5.3 Source IP Concentration
ss -tan state established | awk 'NR>1 {print $5}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head -20Normal: long-tail distribution. Attack: few IPs with thousands of connections each.
5.4 Connection Rate vs Request Rate
# Passive connection rate (15s sample) P1=$(nstat -az | awk '/TcpPassiveOpens/ {print $2}'); sleep 15; P2=$(nstat -az | awk '/TcpPassiveOpens/ {print $2}'); echo $(( (P2-P1)/15 )) conn/s # Request completion rate L1=$(wc -l < /var/log/nginx/access.log); sleep 15; L2=$(wc -l < /var/log/nginx/access.log); echo $(( (L2-L1)/15 )) req/s # Ratio ACTIVE=$(ss -tan state established | tail -n +2 | wc -l); REQ_PER_SEC=...; awk -v a=$ACTIVE -v r=$REQ_PER_SEC 'BEGIN{ if(r>0) printf "ratio=%.1f
", a/r; else print "ratio=undef" }'Ratio > 100 with high reading strongly indicates Slowloris.
5.5 Worker Process State
top -H -b -n 1 -p $(pgrep -f "nginx: worker" | paste -sd, -) | head -30Attack: workers at 1-5% CPU but holding many connections. Also check worker file descriptor usage vs worker_rlimit_nofile.
5.6 Packet Capture Evidence
tcpdump -ni eth0 'tcp port 80' -c 200 -ttSlowloris signature: same TCP session, 10-30s intervals, 1-12 bytes per packet, slow sequence progression.
5.7 Decision Matrix
Fill six evidence rows (state distribution, connection/request ratio, source distribution, worker behavior, packet features, log growth). ≥4 rows pointing to attack + packet evidence = Slowloris conclusion.
6. Command Reference
Categorized cheat sheets for: connection totals, state breakdown, source analysis, rate measurement, packet capture, limits/process checks, kernel counters, bandwidth, conntrack, connection duration.
7. Mitigation & Hardening
7.1 Emergency Principles
Backup rules/configs first ( iptables-save, cp -a /etc/nginx).
Start with minimal granularity (block confirmed IPs first).
Every action must have a rollback command.
Continuous evidence collection during mitigation.
7.2 Per-IP Connection Limit (connlimit)
iptables -I INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 --connlimit-mask 32 -j DROPThreshold from business baseline (99th percentile × 2).
7.3 IP Blocking with ipset
ipset create attack_ip hash:ip timeout 3600; ipset add attack_ip 203.0.113.88; iptables -I INPUT -m set --match-set attack_ip src -j DROPAuto-expires after timeout.
7.4 SYN Rate Limiting (if SYN Flood present)
iptables -I INPUT -p tcp --syn --dport 80 -m hashlimit --hashlimit-above 500/sec --hashlimit-burst 1500 --hashlimit-mode srcip --hashlimit-name syn80 -j DROP7.5 Kernel Tuning
sysctl -w net.core.somaxconn=4096; sysctl -w net.ipv4.tcp_max_syn_backlog=8192; sysctl -w net.ipv4.tcp_syncookies=1; sysctl -w net.ipv4.tcp_synack_retries=1; sysctl -w net.ipv4.tcp_fin_timeout=30; sysctl -w net.ipv4.tcp_abort_on_overflow=1Persist in /etc/sysctl.d/90-nginx-tune.conf.
7.6 Nginx Configuration Hardening
client_header_timeout 10s; client_body_timeout 10s; send_timeout 10s; limit_conn_zone $binary_remote_addr zone=perip:10m; limit_conn perip 100; limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=10r/s; limit_req zone=req_per_ip burst=100 nodelay; keepalive_timeout 65s; worker_rlimit_nofile 65535; worker_connections 16384; listen 80 default_server backlog=4096;Reload with nginx -t && nginx -s reload.
7.7 Upstream Keepalive
upstream backend { server 10.0.0.11:8080; server 10.0.0.12:8080; keepalive 64; } server { location /api/ { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; } }7.8 Capacity Expansion for Normal Traffic
Ordered alignment: fs.file-max → nr_open → worker_rlimit_nofile → systemd/container limits → worker_connections → listen backlog ↔ somaxconn → tcp_max_syn_backlog → upstream keepalive. Validate each step with ss -lnt, ListenOverflows, load testing.
8. Monitoring & Alerting
Key Prometheus metrics: node_netstat_Tcp_CurrEstab, nginx_connections_reading/writing/waiting, nginx_http_requests_total, node_netstat_TcpExt_ListenOverflows. Alert rules based on baseline multiples (e.g., connection/request ratio > 3× baseline for 10m). Grafana panels for established, reading/writing/waiting, ratio, listen overflows, passive open rate.
9. Troubleshooting Paths
Decision tree: high connections → check reading (Slowloris) → check SYN_RECV (SYN Flood) → check waiting (normal keepalive) → check CLOSE_WAIT (upstream issue) → check TIME_WAIT (short connections). Common misjudgments: keep-alive as attack, NAT aggregation as single-IP attack, single-metric decisions.
10. Risk Reminders
Avoid self-inflicted harm (tcpdump CPU, strace overhead, over-blocking).
Never hardcode thresholds; use baselines.
Blocking IPs ≠ resolution; need layered defense.
Reload safely; avoid restart during peaks.
Kernel params have trade-offs; document rationale.
No architectural changes during attack.
Evidence before action.
11. Validation
Post-mitigation re-run exact same evidence commands. For normal traffic expansion, verify new capacity absorbed, queues not overflowing, latency stable. Long-term: weekly baseline reviews, quarterly attack drills, 24/72h post-event monitoring.
12. Rollback Procedures
Precise rollback commands for iptables/ipset, sysctl, Nginx config. Test rollbacks in staging; measure rollback time.
13. Production Practices
Daily baselines in Grafana; refresh after major events/releases.
Change management with impact scope, window, validation, rollback.
Alert consolidation into single "connection anomaly" alert with SOP.
Emergency access ready; annual drills with role separation (decider, executor, reviewer).
14. Summary
State distribution is first watershed: high reading + low requests = Slowloris; high waiting tracking requests = normal.
Connection/request ratio is second watershed: thousands of connections with tens of req/s = attack.
Evidence chain ≥3 lines; decision matrix before action.
Check two charts first: connection vs request correlation, stub_status health.
Mitigation order: backup → minimal stopgap (connlimit/block) → kernel+Nginx time/connection limits → edge protection.
Attack consumes connection quota, not CPU/bandwidth.
Monitoring must be baseline-relative with structural metrics.
Golden rule: When connections spike at midnight, first spend three minutes filling the six evidence rows — not blocking IPs.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
