Skip to content
API Gateway Showdown: Production Tuning, Benchmarks & Migration Fixes for APISIX/Kong/Envoy
Operations 53 min read

API Gateway Showdown: Production Tuning, Benchmarks & Migration Fixes for APISIX/Kong/Envoy

This comprehensive guide compares Apache APISIX, Kong Gateway, and Envoy Proxy across architecture, configuration, performance benchmarks, and real-world troubleshooting — providing production-ready tuning scripts, kernel optimizations, and migration strategies for high-throughput microservice gateways.

MaGe Linux Operations
MaGe Linux Operations
MaGe Linux Operations

1. Architecture Fundamentals: Control Plane & Routing Engine Differences

API gateways handle north-south traffic, auth, rate-limiting, canary routing, and observability. Despite APISIX 3.20.0 and Kong 3.16/3.14 LTS both using OpenResty (NGINX + LuaJIT) on the data plane, while Envoy Proxy 1.39/1.40 uses a pure C++ async event-driven architecture, their production behavior diverges sharply in tail-latency stability, hot-update latency, and operational complexity.

1.1 Core Mechanisms

Routing Algorithm Complexity : Kong (traditional linear/regex traversal) degrades to O(N) scan at thousands of rules; 3.x added expressions router but dynamic changes still rebuild memory tables. APISIX uses lua-resty-radixtree (Radix Tree) for O(K) prefix matching (K = URI path length, independent of route count). Envoy employs C++ compile-time optimized prefix/hash trees with no LuaJIT boundary overhead or GC pauses.

Control-Plane State Sync : Kong polls PostgreSQL (default 5s) — DB becomes bottleneck/failure amplifier; DB-less mode pushes full config via Admin API, causing worker memory spikes at 10k+ routes. APISIX leverages etcd v3 Raft strong consistency + incremental Watch for millisecond-level (<50ms) incremental pushes to worker shared memory without NGINX reload. Envoy uses xDS gRPC streaming (ADS/EDS/RDS) for real-time binary Protobuf config delivery with connection draining.

Cross-Core Scaling & Memory Model : NGINX/OpenResty (Kong/APISIX) uses multi-process Master-Worker model — each worker runs independent LuaJIT VM (2GB addressing limit, cross-process shared-dict lock contention). Envoy uses single-process multi-thread (Worker Thread Per Core) with thread-local storage for near-lock-free concurrency.

1.2 Data-Flow ASCII Diagrams

Two diagrams contrast OpenResty+etcd incremental flow (APISIX) vs C++ multi-thread + xDS control plane (Envoy Gateway), plus Kong's traditional PostgreSQL polling bottleneck path.

[路线 A: OpenResty + etcd 增量配置流 (以 Apache APISIX 为代表)]
+-------------+   etcd Watch (gRPC)   +----------------------+
| etcd 3-Node |-------------------->| APISIX Master/Worker |
| (Raft 存储) |   毫秒级增量配置推送   | (NGINX + LuaJIT VM)  |
+-------------+                     +----------+-----------+
                                         | RadixTree
+-------------+   HTTP/gRPC :443         v O(K) 匹配
| 客户端流量  |----------------------->|  Lua 插件链 -> 后端  |
+-------------+                         +----------------------+

[路线 B: C++ 多线程 + xDS 流推送 (以 Envoy Gateway 为代表)]
+-------------+   xDS (ADS/RDS gRPC)  +----------------------+
| Envoy GW    |----------------------->| Envoy Data Plane     |
| 控制面 Pod  |   Protobuf 增量下发    | (C++ 多线程无锁环)   |
+-------------+                         +----------+-----------+
                                         | FilterChain
+-------------+   HTTP/2 / gRPC :443    v C++ 原生转发
| 客户端流量  |----------------------->|  L7 Filter -> 后端   |
+-------------+                         +----------------------+

[Kong 传统 DB 模式同步路径]
+---------------+  每 5s 全量/增量轮询  +------------------+
| PostgreSQL 16 |<----------------------| Kong Data Plane  |
| (单点写瓶颈)  |  连接数 = 节点 × 核数 | (Worker Cache)   |
+---------------+                       +--------+---------+
                                            | Expressions
+---------------+  HTTP :8000              v 路由表重建
| 客户端流量    |----------------------->| Lua 插件链 ->后端 |
+---------------+                         +------------------+

2. Multi-Dimensional Selection Matrix

2.1 Architecture & Ecosystem Comparison

Routing Engine : APISIX — RadixTree O(K), no degradation at 10k+ routes; Kong — Expressions tree, higher regex overhead; Envoy — C++ prefix/exact match, zero GC jitter.

Config Storage & Hot Update : APISIX — etcd Watch millisecond hot update; Kong — PostgreSQL polling or DB-less push; Envoy — xDS gRPC streaming (no external storage).

Multi-Language Plugins : APISIX — Lua/Go/Python/Wasm; Kong — Lua/Go/Python/JS; Envoy — C++ native Filter/Wasm/ExtProc.

AI/MCP Gateway : APISIX — Built-in AI proxy, semantic cache, MCP routing; Kong — Enterprise edition or community plugins; Envoy — Envoy AI Gateway (Agent Router).

2.2 Hardware, Licensing & Operational Cost

Control-Plane Dependencies : APISIX — 3-node etcd cluster; Kong — HA PostgreSQL for DB mode; Envoy — No external DB, hard dependency on K8s API Server.

Single-Node Memory Baseline : APISIX — Medium (~250-400 MB); Kong — Higher (LuaJIT dict ~500 MB+); Envoy — Minimal (C++ process ~80-150 MB).

License & Vendor Lock-in : APISIX — Apache 2.0 (ASF, no lock-in); Kong — Core Apache 2.0, advanced rate-limiting/security in Enterprise; Envoy — Apache 2.0 (CNCF graduated, fully free).

First-Line Troubleshooting Barrier : APISIX — Medium (NGINX/Lua knowledge); Kong — Lower (mature Admin API & docs); Envoy — Higher (xDS topology & C++ stats).

2.3 Selection Red Lines

If team cannot operate etcd or not on K8s, but wants zero external deps for lightweight sidecar/north-south proxy → exclude DB-mode Kong; prefer Envoy or APISIX Standalone (YAML mode).

If business gateway needs frequent dynamic route changes (hundreds of daily canary releases by 100+ devs) with millisecond effect and zero connection flap → APISIX is current top recommendation.

If cluster fully embraces Kubernetes Gateway API with heavy gRPC/HTTP3/service-mesh east-west traffic → Envoy Gateway is optimal long-term evolution.

3. Recommended Solution: APISIX 3.20 Full Production Deployment

3.1 Hardware Baseline

3 gateway data-plane nodes (8 vCPU/16 GiB/10GbE), 3 dedicated etcd nodes (4 vCPU/8 GiB/NVMe), 4 Go HTTP/2 echo backends (8 vCPU/16 GiB), all on Rocky Linux 9.4 (Kernel 5.14). Tested versions: APISIX 3.20.0, Kong 3.14 LTS, Envoy 1.39.3.

3.2 Linux Kernel Tuning ( /etc/sysctl.d/99-gateway-prod.conf )

# 1. Listen queue & SYN backlog protection
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.core.netdev_max_backlog = 32768

# 2. Upstream short-connection port exhaustion & TIME_WAIT recycle
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# 3. Socket buffers & BBR congestion control
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# 4. Connection tracking table limit
net.netfilter.nf_conntrack_max = 1048576

Apply with sysctl --system (no restart). Verify NIC multi-queue RSS alignment with ethtool -l eth0 (should show 8 combined queues for 8 vCPU).

3.3 APISIX Production config.yaml

Key sections: dual HTTP/HTTPS listeners (9080/9443) with TLS 1.2/1.3, HTTP/2 enabled, router.http: radixtree_host_uri, deployment.role: traditional with etcd provider, Admin API restricted to management subnet (10.30.10.0/24) with admin_key from env var, etcd TLS config with 3-node endpoints, NGINX worker settings ( worker_processes: auto, worker_connections: 65535, worker_rlimit_nofile: 1048576), upstream keepalive (1024 connections, 10000 requests, 60s timeout), shared dicts for plugins (128m each for limit-req/limit-count, 64m for prometheus), Prometheus export on internal IP:9091.

Systemd override sets LimitNOFILE=1048576, LimitCORE=0, and loads /etc/sysconfig/apisix.env (0600). /etc/security/limits.d/99-apisix.conf mirrors nofile limits.

3.4 Dynamic TLS & Routing via Admin API

Certificates deployed via PUT to /apisix/admin/ssls/1 with SNI list — milliseconds to all workers, no nginx -s reload. Upstream with active health checks ( /healthz, 2s interval, 2 successes healthy, 1s/3 failures unhealthy). Route with limit-count (5000 req/s per remote_addr) and Prometheus labels. Global request-id plugin injects X-Request-ID. proxy-mirror at 5% sample for canary validation.

3.5 LLM SSE Streaming Config

Route /v1/chat/completions disables response buffering via proxy-rewrite setting X-Accel-Buffering: no, extends read timeout to 300s. Verified with curl -N showing immediate token streaming.

3.6 Four-Dimensional Config Breakdown

① Must-Change (pre-launch) : admin_listen.ip, prometheus.export_addr.ip to node's management IP; etcd.host to actual TLS endpoints; APISIX_ADMIN_KEY strong random 32+ chars in env file (0600).

② Coupled Parameters : Upstream keepalive (1024) + keepalive_requests (10000) must align with ip_local_port_range (~55k ports) to avoid exhaustion. Shared dicts for limit-req/limit-count (128m) sized for distinct client IP/token cardinality — exhaustion causes 500. Prometheus dict (64m) must avoid high-cardinality labels (dynamic URI as label).

③ Performance Formulas : worker_processes = auto (equals logical CPU cores = 8). Max theoretical concurrent connections = workers × worker_connections / 2 ≈ 262k (each proxied request consumes 2 connections). Upstream keepalive per backend = workers × upstream.keepalive; set 512-1024 if backend handles 8k concurrency. access_log_buffer: 16384 cuts write() syscalls by 90%+.

④ Absolute Red Lines : Never set allow_admin: 0.0.0.0/0 with disabled admin_key (public route rewrite + Lua injection). Never run error_log_level: debug long-term (disk fill, 70%+ throughput drop). Never change deployment.etcd.prefix live (instant 404 on all routes).

3.7 Operational Commands & PromQL Alerts

Key commands: route/etcd sync verification, ss -tan connection state analysis (TIME-WAIT >30k indicates missing upstream keepalive), etcd storage health ( endpoint status), nf_conntrack usage (<70%), TLS/HTTP2 ALPN verification, Envoy admin port cluster health, and PromQL rules for 5xx rate (>1%), P99 latency (>200ms), gateway-only P99 (>15ms), etcd reachability.

4. Alternative Configurations & Migration Pitfalls

4.1 Kong DB-less Declarative Config

Set database=off, declarative_config=/etc/kong/kong.yml, router_flavor=expressions, upstream keepalive 512/10000, mem_cache_size 256m. kong.yml defines upstreams with active health checks, services, routes, and rate-limiting plugin. Validate with kong config parse. For SSE, disable nginx_proxy_proxy_buffering and nginx_proxy_proxy_request_buffering. Migration pitfalls: DB-mode upgrade requires kong migrations up/finish — run on standby first; router flavor change from traditional_compatible to expressions requires regex rewrite (capture groups → new DSL).

4.2 Envoy Gateway (K8s) & Standalone

Gateway API resources: HTTPRoute with weighted backendRefs for canary. BackendTrafficPolicy for circuit breaker (maxConnections 4096, maxPendingRequests 2048) and timeouts. EnvoyProxy CRD binds concurrency (8) to container CPU limits (requests 4, limits 8) to avoid CFS throttling. Standalone envoy.yaml configures clusters with circuit breakers, outlier detection (consecutive 5xx=5, 5s interval, 30s base ejection, 50% max), and strict DNS round-robin. Validate with envoy --mode validate. PromQL alerts for upstream 5xx, P99 latency, connection pool overflow. For LLM SSE, BackendTrafficPolicy sets requestTimeout: 300s, streamIdleTimeout: 120s. Migration pitfall: default concurrency reads host CPU (e.g., 64 cores) but container limit is 4 → 64 threads thrash CFS quota, P99 spikes 6x. Fix: explicit concurrency: 4 in EnvoyProxy CRD or --concurrency 4 flag.

5. Identical-Hardware Benchmark Results

5.1 Quantitative Comparison (8 vCPU/16 GiB, 2000 routes, 5-min steady state)

Metric                          APISIX 3.20      Kong OSS 3.14 (DB-less)   Envoy 1.39 (8 Threads)
Peak Throughput (QPS)           118,400          89,200                    112,600
50K RPS Steady P99 Latency      3.8 ms           8.4 ms                    3.1 ms
1000-Route Full Hot-Update Time 42 ms (etcd Watch) 680 ms (full POST rebuild) 65 ms (xDS incremental)
Steady Memory (2000 routes)     340 MB           580 MB                    165 MB

Raw HdrHistogram at 50K RPS: APISIX P99 3.80ms, Kong 8.40ms, Envoy 3.10ms. CPU sampling ( mpstat -P ALL) shows balanced softirq across cores (RSS working).

5.2 Key Takeaways

Memory & Tail Latency Extreme : Envoy wins with C++ async zero-GC — 3.1ms P99, 165MB RAM — ideal for resource-constrained sidecars or massive K8s Ingress.

Throughput & Hot-Update Balance : APISIX delivers 118k QPS with 42ms incremental updates via etcd, far faster than Kong's 680ms full reload (which causes seconds of 100% worker CPU spikes at 10k+ routes).

6. Three Real-World Troubleshooting Cases

Case 1: Kong DB Connection Storm During Scale-Out

Symptom : Scaling from 6 to 30 nodes during promo — all nodes latency spike, health checks fail (502), PostgreSQL CPU 100%, 1500+ active connections.

Root Cause : Default 5s db_update_frequency + 30 nodes × 16 workers = 480 concurrent full-table paginated scans ( SELECT * FROM routes ORDER BY id LIMIT 1000 OFFSET 2000) — DB meltdown, timeouts trigger Kong retries → death spiral.

Fix : Emergency switch to Hybrid Mode (CP/DP separation) or DB-less; stopgap: db_update_frequency=60, rate-limit scale-out.

Before vs After : PG connections 1140+ → 0; cold-start ready >6 min → 12s; config propagation P99 15-30s → <1s.

Case 2: APISIX/Envoy Co-Location Regex Backtracking P99 Spike

Symptom : P50 2.1ms but P99 periodically 180ms; gateway-only latency ( type=apisix) 160ms, backend <3ms.

Root Cause : Route /api/v1/promo/(.*)/(.*)/detail with greedy nested (.*) triggered PCRE catastrophic backtracking on malformed long URLs. OpenResty single-threaded worker event loop blocked — all concurrent requests on that worker queued.

Fix : Rewrite to RadixTree param path /api/v1/promo/:act_id/:channel/detail; enable pcre_jit on and limit backtrack steps.

Before vs After : Gateway P99 162ms → 1.4ms; single-worker CPU 100% → 24% balanced; 50K RPS error rate 0.34% → 0%.

Case 3: Envoy CPU Throttling from Concurrency/CPU-Limit Mismatch

Symptom : 15K RPS, P99 3ms → 22ms; CPU not saturated but container_cpu_cfs_throttled_periods_total alerting.

Root Cause : Pod limit 4 CPU, but Envoy read host /proc/cpuinfo (64 cores) and spawned 64 worker threads. CFS quota (400ms per 100ms) exhausted in milliseconds → 79% periods throttled, 20ms+ periodic tail latency.

Fix : Set concurrency: 4 in EnvoyProxy CRD (1:1 with CPU limit) and roll restart.

Before vs After : P99 22.4ms → 2.9ms; throttled periods 79% → <0.5%; RSS 410MB → 115MB.

7. Automation Scripts & Risk Matrix

7.1 Unified Benchmark Runner ( gw-bench-runner.sh )

Locks threads=8, connections=512, duration=60s, warmup 10s, outputs standardized latency percentiles and error counts for apples-to-apples comparison.

7.2 APISIX Canary Weight Switch Script

Reads APISIX_ADMIN_KEY from secured env file (0600), backs up current route, validates weight sum=100, supports --dry-run, requires interactive yes confirmation before PATCH.

7.3 Pre-Flight 7-Step Checklist

apisix test

syntax validation

etcd cluster health ( endpoint health)

Worker Max open files = 1048576

Management port 9180 bound only to internal IP

Prometheus metrics endpoint serving route counters

End-to-end request returns X-Request-ID, no Server header leak

etcd snapshot backup ( etcdctl snapshot save)

7.4 Production Change Risk Matrix

Modify config.yaml / kong.conf + reload : Memory briefly doubles during worker transition. Pre-check: apisix test / kong check; drain SLB weight per node. Rollback: Restore config + reload.

Global plugin (rate-limit/WAF) rollout : Misconfigured rule → site-wide 403/429. Pre-check: Enable on single non-critical test route + dry-run/log mode 24h. Rollback: Admin API DELETE/PATCH plugin.

Kernel sysctl network params : Instantly affects all host TCP connections. Pre-check: Low-traffic single node; verify ss -s & dmesg. Rollback: Restore old /etc/sysctl.d/ + sysctl --system.

Cross-version gateway upgrade (e.g., APISIX 3.16→3.20) : Auth boundary behavior changes possible. Pre-check: etcdctl snapshot save; upgrade 1 canary node at 5% traffic. Rollback: Rollback binary + etcdctl snapshot restore.

8. Summary Conclusions

No Silver Bullet — Context Match : APISIX for dynamic 10k+ route hot updates, AI/MCP ecosystem, pure Apache 2.0; Envoy for minimal memory, K8s Gateway API native, gRPC mesh unification; Kong only with DB-less/Hybrid CP-DP to eliminate PostgreSQL coupling.

Tune OS First, Then Gateway : Upstream keepalive + ip_local_port_range / somaxconn /BBR often beats adding gateway nodes for P99 reduction.

Beware Regex Backtracking & Blocking Plugins : Never nest (.*) on hot paths; separate gateway-only vs backend latency via Prometheus for precise bottleneck isolation.

Control-Plane Storage Governance Upfront : etcd auto-compaction ( auto-compaction-retention: 1h) + 8GB quota ( quota-backend-bytes: 8589934592) — default 2GB fills fast under frequent canary releases, locking entire routing plane read-only.

Container Thread/Process Count Must Match cgroup Quota : Explicit --concurrency (Envoy) or worker_processes (OpenResty) = container CPU limit to prevent CFS throttling tail latency.

Streaming Inference Routes Must Disable Response Buffering : LLM/Agent /v1/chat/completions requires proxy_buffering off + read timeout ≥300s to avoid TTFT degradation.

Syntax Check + Snapshot Backup Dual Insurance : Every static config reload preceded by apisix test / kong config parse / envoy --mode validate; every control-plane upgrade backed by etcdctl snapshot or declarative YAML.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

API GatewayPerformance TuningTroubleshootingBenchmarkingEnvoyAPISIXKongProduction Operations
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.