Nginx Rate Limiting in Practice: Defending Against CC Attacks and Traffic Spikes

Problem Background

Nginx is the de facto ingress layer for most internet companies but lacks built-in rate limiting. Without it, CC attacks, traffic spikes, crawlers, or buggy client retries can cause CPU saturation, connection exhaustion, bandwidth depletion, and cascading failures. Firewall IP blocking is reactive; WAF is a separate product line. Nginx's native rate limiting modules suffice for most small-to-medium scenarios.

Core capabilities come from three directives: limit_req_zone, limit_conn_zone, limit_req, complemented by geo for black/white lists, map for request identification, and OpenResty for complex logic.

Applicable Scenarios

Ingress-layer CC attack protection

Traffic spike shaving (flash sales, limited-time events)

Crawler and malicious request blocking

Per-IP / per-user / per-interface rate limiting

Fine-grained API limiting (login, register, order, payment)

API gateway, microservice, CDN origin, canary release, SDK retry, database write, SMS/email, captcha/token, file download, mobile weak network, cross-region traffic shaping

Core Concepts

The Three Musketeers of Nginx Rate Limiting

limit_req_zone

Request rate limiting based on a fixed time window.

limit_req_zone $key zone=name:size rate=rate;

$key

: limiting key, commonly $binary_remote_addr (by IP), $server_name (by domain), or custom variables. zone: shared memory zone in name:size format. rate: rate limit, e.g., 10r/s (10 requests/second) or 60r/m (60 requests/minute).

Underlying algorithm: leaky bucket — tokens fill at a fixed rate, each request consumes a token; no token means reject or queue.

limit_req

Applies the limiting rule in a specific location or server.

limit_req zone=name [burst=number] [nodelay | delay=number];

zone

: references the limit_req_zone name. burst: allowed burst requests; excess requests queue. nodelay: excess requests return immediately without queuing. delay: number of excess requests to delay before queuing.

Note: burst is queue capacity, not total burst capacity.

limit_conn_zone / limit_conn

Connection count limiting, controlling concurrent connections per key.

limit_conn_zone $key zone=name:size; limit_conn zone_name number;

Suitable for limiting long connections and slow requests.

Status Code Directives

limit_req_status 429; limit_conn_status 429;

Default is 503; 429 allows clients to recognize and back off.

Log Level

limit_req_log_level warn; limit_conn_log_level warn;

Default error; production recommends warn.

Leaky Bucket Algorithm

Nginx's limit_req implements leaky bucket:

Tokens fill at rate into bucket Bucket capacity = burst Request consumes one token Empty bucket → request rejected (or queued)

Example: rate=10r/s burst=20 nodelay Bucket normally has 0 tokens.

Burst of 20 requests all pass.

Subsequent requests wait 100ms per token.

21st+ requests rejected immediately.

Difference from token bucket: leaky bucket enforces constant output rate (good for smoothing), token bucket allows bursts (good for jitter tolerance).

burst / nodelay / delay Differences

No burst : Strict rate, excess rejected immediately — strict limiting.

burst=N : Allows N requests to queue — allows jitter.

burst=N nodelay : N requests pass immediately, excess rejected — allows burst but no queue.

burst=N delay=M : First M pass immediately, M-N queue, beyond rejected — compromise.

Concrete examples show how each configuration behaves under load.

Key Selection

$binary_remote_addr

is most common (IP-based), but scenarios vary:

By IP: $binary_remote_addr — most common, saves memory.

By IP + URL: $binary_remote_addr$request_uri — prevents single IP scanning multiple URIs.

By user: $http_authorization (extract user ID) — post-login.

By domain: $server_name — prevents specific domain attacks.

By URI: $request_uri — prevents single URI abuse.

By IP+UA: Custom combination — prevents proxy IP rotation.

Shared memory size relates to key count: 1MB ≈ 16K IPs; exceeding zone size triggers LRU eviction.

Common Pitfalls

Configuring rate limiting makes it effective. Wrong — if rate/burst too large, limiting does nothing. Must load-test and adjust.

Rate limiting doesn't drop requests. Wrong — excess requests are rejected (unless queued via burst). Clients must handle 429 with retry/backoff.

One-time configuration. Wrong — parameters need continuous adjustment as business grows.

Rate limit all interfaces. Wrong — prioritize write-heavy, resource-intensive endpoints; read-heavy cached endpoints lower priority.

Rate limiting stops all attacks. Wrong — distributed attacks (many IPs) need WAF, captchas, IP reputation.

Overall Investigation & Implementation Approach

Five Layers of Rate Limiting

1. Client-side (SDK retry backoff, button debounce) 2. Ingress layer (Nginx / CDN / WAF) 3. Application layer (Sentinel / Guava RateLimiter / custom) 4. Middleware layer (Redis token bucket / MQ limiting) 5. Database layer (connection pool / slow SQL interception)

Every layer must be implemented.

Parameter Design Steps

Business baseline assessment : obtain normal-period QPS from monitoring.

Peak assessment : estimate peak at 3-5x business volume.

Burst assessment : estimate burst at 1.5-2x peak.

Per-interface assessment : login, order, query evaluated separately.

Per-user assessment : VIP vs regular users separated.

Implementation Order

Assess first : use monitoring/logs for real data.

Then load test : with wrk, ab, vegeta.

Then configure : set limit_req based on test results.

Canary deploy : enable on subset of nodes.

Observe : watch 429 ratio, client impact.

Full rollout : expand to all nodes.

Troubleshooting Steps for Rate Limiting Anomalies

1. 429 spike → check limit_req config, burst setting 2. False positives → check key selection, rate too strict 3. Upstream QPS unchanged → verify limit_req actually applied 4. Client retry storm → client must implement exponential backoff 5. Memory exhaustion → check zone size adequacy

Practical Steps

5.1 Basic Rate Limiting: Per IP

Define zone in http block, apply in location, test config, reload, load test with ab.

5.2 Multi-level Limiting: Per Interface

Separate zones for login (1r/s), register (1r/s), query (50r/s), order (5r/s), global fallback (100r/s), each with appropriate burst.

5.3 Multi-dimensional: IP + URI

Zone key $binary_remote_addr$request_uri, zone size increased (20MB ≈ 320K combos).

5.4 Rate Limiting + Whitelist (IP bypass)

Use geo to mark internal networks, map to convert to empty key for whitelist. Improved version uses map for safer key transformation.

5.5 Rate Limiting + Blacklist (specific IP deny)

geo

marks bad IPs, if ($is_black) { return 444; } silently drops connection.

5.6 CC Attack Defense: UA Identification + Rate Limiting

map

matches known attack tool UAs (sqlmap, nmap, scrapy, etc.) and empty UA, returns 444. Caution: may block legitimate clients without UA.

5.7 Traffic Spike Shaving: Phased Rate Limiting

Define normal and peak zones, switch via map $time_hour $limit_zone (avoids "evil if").

5.8 Connection Limiting: Slow Attack Prevention

limit_conn_zone

per IP (10) and per server_name (1000).

5.9 Rate Limiting + Bandwidth Throttling

limit_rate 100k;

per connection, limit_rate_after 1m; start throttling after 1MB.

5.10 Rate Limiting Logs & Monitoring

Custom log format includes $limit_req_status, $limit_conn_status, $request_time, upstream metrics. Note: $limit_req_status requires Nginx 1.17.7+.

Common Commands

6.1 Nginx Rate Limiting Config

nginx -t nginx -s reload nginx -T | grep -E 'limit_req|limit_conn' tail -F /var/log/nginx/error.log | grep -i limit awk '$9 == 429' /var/log/nginx/access.log | wc -l awk '$9 == 429' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20 awk '$9 == 429' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

6.2 Load Testing Tools

ab (Apache Bench)

ab -n 1000 -c 10 'http://api.example.com/test' ab -t 60 -c 100 'http://api.example.com/test'

wrk

wrk -t 4 -c 100 -d 30s 'http://api.example.com/test' wrk -t 4 -c 100 -d 30s --latency 'http://api.example.com/test' wrk -t 4 -c 100 -d 30s -s post.lua 'http://api.example.com/api/post'

vegeta

echo "GET http://api.example.com/test" > targets.txt vegeta attack -duration=30s -rate=1000 -targets=targets.txt | vegeta report

hey

hey -n 1000 -c 10 'http://api.example.com/test'

6.3 IP Frequency Statistics

awk '{print $1}' /var/log/nginx/access.log | tail -10000 | sort | uniq -c | sort -rn | head -20 # Top IP last hour, empty UA detection, per-second >100 requests detection

6.4 Firewall Integration (iptables / nftables)

Collect high-frequency IPs, add to iptables. For many IPs, use ipset for efficiency.

6.5 Temporary IP Blocking (CC Emergency)

Bash script scans recent logs, adds offending IPs to iptables, saves rules.

Configuration Examples

7.1 Complete Production Rate Limiting Config

Full nginx.conf with geo black/white lists, UA detection, multiple zones (global, API, login, static), connection limits, 429 status, log format, upstream with health checks, proxy settings.

7.2 OpenResty + Redis Distributed Rate Limiting

Lua script uses Redis sorted set sliding window (1 min, 100 requests). Falls through if Redis unavailable.

7.3 Tengine Active Health Checks

check

directive with interval, rise/fall counts, HTTP health check.

7.4 Rate Limiting Fallback Page

error_page 429 /errors/429.html;

with internal location serving friendly HTML.

7.5 Rate Limiting + JSON Response

error_page 429 = @rate_limit;

named location returns JSON with retry_after.

Log & Metrics Observation

8.1 Key Logs

Nginx error log shows limiting requests, excess: 20.000 by zone "api_limit". Custom access log includes limit status variables.

8.2 Key Metrics

429 ratio

Per-URI 429 ratio

Per-IP 429 count

Rate limiting error log count

limit_req zone memory usage

8.3 Prometheus Scraping

Configure nginx-exporter target.

8.4 Key PromQL Queries

# 429 ratio sum(rate(nginx_http_requests_total{status="429"}[5m])) / sum(rate(nginx_http_requests_total[5m])) # Per-URI 429 rate sum by (uri) (rate(nginx_http_requests_total{status="429"}[5m])) # Per-IP 429 count sum by (client_ip) (rate(nginx_http_requests_total{status="429"}[5m]))

8.5 Alerting Rules

Alert on 429 spike (>100/min), 429 ratio >5%, active connections >50000. Thresholds must be tuned to business baseline.

Troubleshooting Paths

9.1 Case 1: 429 Spike

Symptoms: alert on 429 surge, login failures. Hypotheses: config too strict, traffic spike, client retry storm. Commands to check 429 ratio, top IP/URI, time distribution, error logs. Root cause: single IP high freq → client retry loop; single URI high freq → endpoint scraping. Fixes: adjust rate short-term, client backoff mid-term, distributed limiting long-term. Verify 429 drop, business recovery, client logs. Rollback: comment limit_req, reload.

9.2 Case 2: CC Attack

Symptoms: bandwidth saturation, upstream QPS spike, service slowness. Hypotheses: CC attack, crawler rampage. Commands: top IP, top UA, request time distribution, anomalous UA detection. Root cause: concentrated IPs, attack tool UAs. Fixes: enable UA blocking, iptables blacklist, strict rate limiting. Verify attack IP reduction, 5xx drop, bandwidth recovery. Rollback: widen rate, remove partial blacklist.

9.3 Case 3: Traffic Spike

Symptoms: 5xx spike, flash sale start. Hypotheses: traffic exceeds capacity, upstream slow. Commands: QPS change, 5xx ratio, upstream response time. Root cause: 10x traffic surge, limiting not enabled or burst too large. Fixes: enable strict limiting immediately, scale upstream, client degradation. Verify 5xx drop, business recovery. Rollback: widen limiting post-event, canary restore.

Risk Reminders

10.1 Configuration Risks

Rate too small → false positives

Rate too large → ineffective

Burst too large → burst passes

Burst too small → legitimate requests rejected

Zone too small → OOM or key eviction

Wrong key → wrong limiting dimension

10.2 Change Risks

Load test before changing limit_req Canary deploy

Quick rollback on false positives

Coordinate with client retry backoff

10.3 Upstream Interaction

Rate limiting doesn't speed up upstream

Blocked requests must be visible to client

Blocked requests must be logged

Blocked requests need fallback page

10.4 Monitoring

Monitor 429 ratio

Monitor rate limiting error logs

Monitor upstream QPS

Monitor client retry ratio

10.5 Security

Rate limiting ≠ WAF

Rate limiting ≠ IP blacklist

Rate limiting ≠ captcha

Rate limiting ≠ HTTPS

Validation Methods

11.1 Config Validation

nginx -t wrk -t 4 -c 100 -d 30s 'http://api.example.com/test' # Check 429 ratio from logs

11.2 Canary Validation

Enable on single node, sustained load test 60s, verify 200 + some 429, observe 5 minutes, then expand.

11.3 Canary Rollout Process

Prepare config file, load on test node, continuous observation via log tail, after 5 minutes push to all nodes via config center/Ansible, continue monitoring.

Rollback Plans

12.1 Config Rollback

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.$(date +%Y%m%d_%H%M%S) nginx -t && nginx -s rollback # On issue: restore backup, test, reload

12.2 Disable Rate Limiting

sed -i 's/^    limit_req/# &/' /etc/nginx/nginx.conf nginx -t && nginx -s reload

12.3 Adjust Parameters

Modify rate in limit_req_zone, test, reload.

12.4 Emergency

Restore backup immediately, verify with curl, notify business.

Production Considerations

13.1 Parameter Tuning

Set from monitoring data + business volume

Validate with load test data

Adjust burst from real data

Continuously observe and adjust with growth

13.2 Client Cooperation

Client handles 429 backoff

Client timeout control

Client rate limit UI hints

Client degradation

13.3 Upstream Cooperation

Upstream degradation

Upstream caching

Upstream application-layer limiting

Upstream circuit breaking

13.4 Monitoring

429 ratio

Rate limiting error logs

Upstream QPS

Client retry ratio

13.5 Canary

Single node

Partial traffic

Full rollout

Continuous observation

13.6 Security

WAF integration

IP blacklist

Captcha

HTTPS

13.7 High Availability

Per-datacenter independent limiting

Per-cluster independent limiting

Global limiting via OpenResty+Redis

Dynamic limiting auto-adjusted by business volume

13.8 Capacity

Capacity planning

Elastic scaling

Emergency plans

Chaos drills

Summary

Nginx rate limiting is essential for ingress layer. This article covers:

Fundamentals : limit_req_zone, limit_req, limit_conn_zone, limit_conn; burst/nodelay/delay differences; leaky bucket; key selection.

Practical Configs : per-IP, per-URI, multi-level, whitelist/blacklist, CC defense, spike shaving, OpenResty+Redis.

Production Practices : load test validation, canary deployment, monitoring/alerting, rollback plans.

Key Parameters : rate = real business + buffer; burst = allowed burst + queue tolerance; zone = key count × 16KB.

Multi-layer Defense : Nginx ingress, application (Sentinel/Guava), middleware (Redis token bucket), database (connection pool).

Interview Points : algorithms (leaky vs token bucket), config (limit_req+burst+nodelay), parameters (rate, burst, key), scenarios (CC, spike, crawler), relation to degradation/circuit breaking.

Final acceptance criteria:

On CC attack alert, can deploy protection within 10 minutes.

On 429 spike alert, can distinguish config issue vs client issue.

On tuning request, can calculate reasonable rate/burst from business volume.