Skip to content
Nginx High Connections: Normal Traffic or Slowloris Attack? Diagnostic Guide
Operations 79 min read

Nginx High Connections: Normal Traffic or Slowloris Attack? Diagnostic Guide

This guide provides a systematic forensic process to distinguish between normal high Nginx connections and slow connection attacks (Slowloris), using Linux commands, Nginx stub_status, connection state analysis, rate comparisons, packet capture, and evidence-based decision matrices for accurate diagnosis and mitigation.

Raymond Ops
Raymond Ops
Raymond Ops

1. Problem Background

Operations staff observed thousands of established connections via ss -s while CPU remained under 10% and bandwidth low. The team debated whether this was a Slowloris attack or normal traffic surge, with conflicting suggestions to increase worker_connections, block IPs, or restart Nginx. The core challenge is establishing a forensic process to differentiate "normal high connections" from "slow connection attacks" before taking action.

2. Applicable Scenarios

Suitable when:

Connection count significantly exceeds baseline via ss / netstat and needs classification.

Site shows "connects but won't load" symptoms with low server utilization.

Monitoring alerts on active connections persist.

Planning rate limiting or hardening and need to understand restrictions and risks.

Direct connections reach Nginx bypassing CDN/WAF.

Not suitable when: CPU at 100% (different troubleshooting path), massive SYN_RECV (SYN Flood), Layer 4 load balancer in front (must identify correct layer), or massive short-connection workloads where high connections are normal.

3. Core Knowledge

3.1 Nginx Connection States

Nginx stub_status divides connections into three categories: reading: Reading client request headers/body. writing: Writing response to client. waiting: Keep-alive idle connections awaiting next request.

Sum equals active connections. Slowloris attacks pile up in reading; normal high concurrency grows waiting.

3.2 TCP State Machine Quick Reference

Key states via ss: ESTABLISHED: Both normal and attack connections appear here; must inspect internal activity. SYN_RECV: High counts indicate SYN Flood or small backlog. TIME_WAIT: Normal in short-connection scenarios. CLOSE_WAIT: Often indicates upstream closed but Nginx hasn't.

Critical: ESTABLISHED ≠ "processing request". Keep-alive, stalled, and attack connections all show as ESTABLISHED.

3.3 Slowloris Variants

Slowloris (slow headers): Sends header lines one by one with long delays.

Slow Body / RUDY: Completes headers but sends body extremely slowly.

Slow Read: Sends request but reads response very slowly.

Common trait: Nginx request processing rate (QPS) very low while connection occupancy rate very high. Normal traffic shows opposite pattern.

3.4 Nginx Event Model & Connection Limits

Nginx uses epoll. Total connection capacity ≈ worker_processes × worker_connections. Three-tier limit hierarchy:

Nginx per-worker limit ( worker_connections)

Process file descriptor limit ( ulimit -n / worker_rlimit_nofile)

Kernel global limit ( fs.file-max / fs.nr_open)

All three must align; raising only worker_connections without adjusting others is ineffective.

3.5 Why "High Connections, Low CPU, Low Bandwidth" = Slowloris

Comparison of metrics:

Metric                  Normal High Traffic                    Slowloris Attack
Established connections High, correlates with request rate       High or slowly climbing, decoupled from request rate
CPU                     Fluctuates with load                     Persistently low (workers waiting on read)
Bandwidth               Rises with connections                   Minimal
Access log              Lines continuously generated             Attack connections produce no log entries (requests incomplete)
stub_status distribution waiting/reading follow business rhythm  reading persistently high, waiting minimal
Per-IP connections      Natural distribution (single digits to tens) Many IPs with dozens/hundreds each
New connection rate     Matches request arrival rate             Far exceeds request completion rate

At least three matching evidence lines required for conclusion.

3.6 Process Attribution

System-wide ss -s includes non-Nginx connections. Must attribute established connections to processes (requires root):

ss -tanp | awk 'NR>1 && /ESTAB/ { n=split($NF,a,"="); gsub(/[")]/,"",a[2]); proc=(a[2]?a[2]:"unknown"); cnt[proc]++ } END { for (p in cnt) print cnt[p], p }' | sort -rn | head

Expected output shows nginx dominating. Also separate inbound (port 80/443) vs outbound (upstream) connections.

4. Overall Judgment Framework

Seven-step evidence chain:

Total count & state distribution

Reading/writing/waiting breakdown

Source IP aggregation & concentration

New connection rate vs request completion rate

Access log growth & error codes

Packet capture sampling

Decision matrix conclusion

Principle: Evidence first, then action; every action must be reversible.

5. Practical Steps (with Commands)

5.1 Total Count & State Distribution

ss -s

Then state breakdown:

ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

5.2 Nginx stub_status Analysis

Enable stub_status (internal only), then: curl -s http://127.0.0.1/nginx_status Key: Reading high + requests growth low = Slowloris signature. Continuous sampling (10s intervals) reveals trends.

5.3 Source IP Concentration

ss -tan state established | awk 'NR>1 {print $5}' | sed 's/:[0-9]*$//' | sort | uniq -c | sort -rn | head -20

Normal: long-tail distribution. Attack: few IPs with thousands of connections each.

5.4 Connection Rate vs Request Rate

# Passive connection rate (15s sample) P1=$(nstat -az | awk '/TcpPassiveOpens/ {print $2}'); sleep 15; P2=$(nstat -az | awk '/TcpPassiveOpens/ {print $2}'); echo $(( (P2-P1)/15 )) conn/s # Request completion rate L1=$(wc -l < /var/log/nginx/access.log); sleep 15; L2=$(wc -l < /var/log/nginx/access.log); echo $(( (L2-L1)/15 )) req/s # Ratio ACTIVE=$(ss -tan state established | tail -n +2 | wc -l); REQ_PER_SEC=...; awk -v a=$ACTIVE -v r=$REQ_PER_SEC 'BEGIN{ if(r>0) printf "ratio=%.1f
", a/r; else print "ratio=undef" }'

Ratio > 100 with high reading strongly indicates Slowloris.

5.5 Worker Process State

top -H -b -n 1 -p $(pgrep -f "nginx: worker" | paste -sd, -) | head -30

Attack: workers at 1-5% CPU but holding many connections. Also check worker file descriptor usage vs worker_rlimit_nofile.

5.6 Packet Capture Evidence

tcpdump -ni eth0 'tcp port 80' -c 200 -tt

Slowloris signature: same TCP session, 10-30s intervals, 1-12 bytes per packet, slow sequence progression.

5.7 Decision Matrix

Fill six evidence rows (state distribution, connection/request ratio, source distribution, worker behavior, packet features, log growth). ≥4 rows pointing to attack + packet evidence = Slowloris conclusion.

6. Command Reference

Categorized cheat sheets for: connection totals, state breakdown, source analysis, rate measurement, packet capture, limits/process checks, kernel counters, bandwidth, conntrack, connection duration.

7. Mitigation & Hardening

7.1 Emergency Principles

Backup rules/configs first ( iptables-save, cp -a /etc/nginx).

Start with minimal granularity (block confirmed IPs first).

Every action must have a rollback command.

Continuous evidence collection during mitigation.

7.2 Per-IP Connection Limit (connlimit)

iptables -I INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 --connlimit-mask 32 -j DROP

Threshold from business baseline (99th percentile × 2).

7.3 IP Blocking with ipset

ipset create attack_ip hash:ip timeout 3600; ipset add attack_ip 203.0.113.88; iptables -I INPUT -m set --match-set attack_ip src -j DROP

Auto-expires after timeout.

7.4 SYN Rate Limiting (if SYN Flood present)

iptables -I INPUT -p tcp --syn --dport 80 -m hashlimit --hashlimit-above 500/sec --hashlimit-burst 1500 --hashlimit-mode srcip --hashlimit-name syn80 -j DROP

7.5 Kernel Tuning

sysctl -w net.core.somaxconn=4096; sysctl -w net.ipv4.tcp_max_syn_backlog=8192; sysctl -w net.ipv4.tcp_syncookies=1; sysctl -w net.ipv4.tcp_synack_retries=1; sysctl -w net.ipv4.tcp_fin_timeout=30; sysctl -w net.ipv4.tcp_abort_on_overflow=1

Persist in /etc/sysctl.d/90-nginx-tune.conf.

7.6 Nginx Configuration Hardening

client_header_timeout 10s; client_body_timeout 10s; send_timeout 10s; limit_conn_zone $binary_remote_addr zone=perip:10m; limit_conn perip 100; limit_req_zone $binary_remote_addr zone=req_per_ip:10m rate=10r/s; limit_req zone=req_per_ip burst=100 nodelay; keepalive_timeout 65s; worker_rlimit_nofile 65535; worker_connections 16384; listen 80 default_server backlog=4096;

Reload with nginx -t && nginx -s reload.

7.7 Upstream Keepalive

upstream backend { server 10.0.0.11:8080; server 10.0.0.12:8080; keepalive 64; } server { location /api/ { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; } }

7.8 Capacity Expansion for Normal Traffic

Ordered alignment: fs.file-max → nr_open → worker_rlimit_nofile → systemd/container limits → worker_connections → listen backlog ↔ somaxconn → tcp_max_syn_backlog → upstream keepalive. Validate each step with ss -lnt, ListenOverflows, load testing.

8. Monitoring & Alerting

Key Prometheus metrics: node_netstat_Tcp_CurrEstab, nginx_connections_reading/writing/waiting, nginx_http_requests_total, node_netstat_TcpExt_ListenOverflows. Alert rules based on baseline multiples (e.g., connection/request ratio > 3× baseline for 10m). Grafana panels for established, reading/writing/waiting, ratio, listen overflows, passive open rate.

9. Troubleshooting Paths

Decision tree: high connections → check reading (Slowloris) → check SYN_RECV (SYN Flood) → check waiting (normal keepalive) → check CLOSE_WAIT (upstream issue) → check TIME_WAIT (short connections). Common misjudgments: keep-alive as attack, NAT aggregation as single-IP attack, single-metric decisions.

10. Risk Reminders

Avoid self-inflicted harm (tcpdump CPU, strace overhead, over-blocking).

Never hardcode thresholds; use baselines.

Blocking IPs ≠ resolution; need layered defense.

Reload safely; avoid restart during peaks.

Kernel params have trade-offs; document rationale.

No architectural changes during attack.

Evidence before action.

11. Validation

Post-mitigation re-run exact same evidence commands. For normal traffic expansion, verify new capacity absorbed, queues not overflowing, latency stable. Long-term: weekly baseline reviews, quarterly attack drills, 24/72h post-event monitoring.

12. Rollback Procedures

Precise rollback commands for iptables/ipset, sysctl, Nginx config. Test rollbacks in staging; measure rollback time.

13. Production Practices

Daily baselines in Grafana; refresh after major events/releases.

Change management with impact scope, window, validation, rollback.

Alert consolidation into single "connection anomaly" alert with SOP.

Emergency access ready; annual drills with role separation (decider, executor, reviewer).

14. Summary

State distribution is first watershed: high reading + low requests = Slowloris; high waiting tracking requests = normal.

Connection/request ratio is second watershed: thousands of connections with tens of req/s = attack.

Evidence chain ≥3 lines; decision matrix before action.

Check two charts first: connection vs request correlation, stub_status health.

Mitigation order: backup → minimal stopgap (connlimit/block) → kernel+Nginx time/connection limits → edge protection.

Attack consumes connection quota, not CPU/bandwidth.

Monitoring must be baseline-relative with structural metrics.

Golden rule: When connections spike at midnight, first spend three minutes filling the six evidence rows — not blocking IPs.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

operationsperformance tuningcapacity planningNginxnetwork forensicsLinux troubleshootingconnection analysisSlowloris
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.