Kgateway at Billion‑Scale: Architecture, Principles, and Production‑Ready Guide

Why traditional Ingress fails at scale

When a Kubernetes cluster grows to support many teams, environments, frequent gray releases, and high‑peak concurrency, the simple Ingress model becomes a bottleneck. Typical symptoms are:

Routing rules proliferate, requiring cross‑team reviews for every change.

Traffic governance (gray release, mirroring, circuit breaking, rate limiting, retries) is scattered across the gateway, services, sidecars, and business code.

Certificate rotation, configuration changes, or service scaling cause latency spikes and error‑rate surges.

Ingress specifications lack expressive power for multi‑tenant, policy reuse, and cross‑namespace collaboration.

The gateway runs, but there is no systematic capacity model, observability metrics, or failure‑drill mechanism.

Swapping Ingress‑NGINX for Kong, APISIX, Istio Ingress, or Envoy Gateway does not solve the root problem; the solution is to upgrade the underlying model.

Four essential upgrades

Replace Ingress configuration with Gateway API resource modeling.

Move from a single‑point proxy to a separated control‑plane / data‑plane architecture.

Replace a pile of forwarding rules with a full traffic‑governance system.

Upgrade from merely handling requests to providing capacity, elasticity, and observability under high concurrency.

Gateway API engineering significance

GatewayClass

: platform‑level definition of the gateway implementation. Gateway: infrastructure layer defining listeners, entry capabilities, and exposure methods. HTTPRoute / GRPCRoute / TLSRoute: application‑level routing rules.

Policy resources: authentication, rate limiting, timeout, retry, security, and observability capabilities.

Benefits:

Infrastructure teams own entry capabilities without touching business routing details.

Business teams manage their own Route objects, isolated from global gateway configuration.

Security, flow‑control, and audit can be abstracted into independent policy objects.

Routes and policies can be templated, standardized, and validated at admission.

Kgateway core principles

Kgateway is a Kubernetes‑native control plane for Envoy. It performs three key functions:

Interpret Kubernetes and Gateway API resource semantics.

Generate Envoy configuration objects (Listener, Route, Cluster, Endpoint) from those semantics.

Push the dynamic configuration to the Envoy data‑plane via the xDS protocol.

Unlike reload‑based proxies, Kgateway continuously drives the data‑plane through the control plane.

Control‑plane / data‑plane separation

Configuration changes no longer require proxy process restarts.

Data‑plane can scale horizontally without affecting control‑plane semantics.

Engineering gains:

Configuration changes are independent of Envoy restarts.

Data‑plane replicas can be scaled without touching the control plane.

Control plane can perform validation, admission, snapshot caching, and debounce.

Separate capacity planning and fault isolation for control and data planes.

Translation engine workflow (semantic compilation)

Gateway / Route / Policy / Secret / Service / EndpointSlice
→ Kgateway Watcher watches changes
→ Translation Engine builds an intermediate model
→ xDS Snapshot is assembled
→ ADS streams the snapshot to Envoy
→ Envoy hot‑updates listeners, clusters, routes, and TLS contexts

This pipeline explains why Kgateway excels at high‑frequency configuration changes, large‑scale routing orchestration, dynamic service discovery, and multi‑replica consistency.

xDS core objects

LDS (Listener Discovery Service) defines external ports, protocols, and filter chains. Changes affect new port openings, TLS context updates, filter‑chain rebuilds, and SNI routing.

RDS (Route Discovery Service) holds host, path, header matching, and backend selection. Most business changes land here; route count, host aggregation, match‑tree depth, and regex vs. prefix usage directly impact matching performance and memory usage.

CDS (Cluster Discovery Service) describes upstream service clusters, load‑balancing strategy, circuit‑breaker thresholds, health checks, connection pools, and timeouts. Governance (retries, timeouts, circuit breaking) materializes in clusters.

EDS (Endpoint Discovery Service) provides real‑time backend instance lists. In high‑concurrency environments, endpoint churn is often more frequent than route changes, so monitoring endpoint change frequency, snapshot size, and Envoy processing capacity is crucial.

Production‑grade architecture

A typical large‑enterprise topology includes a global DNS/GSLB layer, an L4 load balancer, the Kgateway data‑plane (Envoy) handling TLS termination, authentication, rate limiting, retries, and traffic splitting, followed by per‑domain namespaces with their own HTTPRoute and Policy objects, and a unified observability stack (Prometheus, Loki, Tempo, OpenTelemetry).

Responsibility layers

Entry layer : certificate termination, SNI/Host routing, basic protocol ingress, access logs.

Governance layer : JWT/OAuth2/API‑Key auth, rate limiting, circuit breaking, retries, timeouts, canary and mirroring, header injection, trace context propagation.

Business routing layer : path and method matching, version split, service aggregation, BFF/API façade integration.

Backend execution layer : actual business processing, domain validation, data access, downstream orchestration.

Resource model design

The Gateway belongs to the platform team; HTTPRoute belongs to individual business namespaces. The allowedRoutes field on a Gateway defines which namespaces may attach routes, establishing the first governance boundary.

Best‑practice HTTPRoute example

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-api
  namespace: payment
  labels:
    app: payment
    gateway-access: public
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-system
      sectionName: https-public
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /payment
          headers:
            - name: X-Region
              value: cn-east
      backendRefs:
        - name: payment-service
          port: 8080
          weight: 100
      timeouts:
        request: 5s

Key points:

Routes are split by business domain rather than a monolithic list.

Header, path, and host conditions are expressed together.

Timeouts are declared explicitly in the route.

Explicit gray release example

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-canary
  namespace: payment
spec:
  parentRefs:
    - name: prod-gateway
      namespace: gateway-system
      sectionName: https-public
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /payment
          headers:
            - name: X-Canary
              value: "true"
      backendRefs:
        - name: payment-v2
          port: 8080
          weight: 100
    - matches:
        - path:
            type: PathPrefix
            value: /payment
      backendRefs:
        - name: payment-v1
          port: 8080
          weight: 95
        - name: payment-v2
          port: 8080
          weight: 5

This layout makes the gray‑release intent auditable, reversible, and observable.

High‑concurrency bottlenecks and tuning

Four common bottlenecks at billion‑request scale:

CPU pressure from TLS, compression, filter chains, and log sampling.

Memory pressure from massive route objects, metric label explosion, and large buffers.

Connection pressure from upstream spikes, insufficient connection pools, and TIME_WAIT accumulation.

Control‑plane pressure when rapid Endpoint changes cause xDS push storms.

Envoy thread configuration

Start concurrency near the number of CPU cores, then tune with realistic TLS, routing, and observability load:

gateway:
  proxy:
    envoyConfigOverrides:
      concurrency: 4
      disableWasm: true
      statsFlushInterval: 5s

Thread count is not a magic switch; it must be validated with load‑testing that measures P99 latency, connection reuse, CPU steal, and memory usage.

TrafficPolicy example (connection pool, retries, circuit breaker)

apiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata:
  name: payment-resilience
  namespace: payment
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: payment-api
  retry:
    retryOn: connect-failure,refused-stream,unavailable,5xx
    numRetries: 2
    perTryTimeout: 1500ms
  timeout:
    requestTimeout: 5s
    idleTimeout: 60s
  circuitBreaker:
    maxConnections: 2000
    maxPendingRequests: 4000
    maxRequests: 8000
    maxRetries: 200
  connectionPool:
    tcp:
      maxConnections: 2000
      connectTimeout: 2s
    http:
      http1MaxPendingRequests: 2048
      http2MaxRequests: 4096
      maxConcurrentStreams: 128

These settings illustrate a production‑grade policy: retries are bounded, timeouts are shorter than upstream budgets, and circuit‑breaker thresholds match backend capacity.

Three‑tier rate limiting hierarchy

Global entry‑point limit protects the gateway itself.

Tenant/application limit prevents a single tenant from exhausting shared resources.

Route‑level limit safeguards sensitive endpoints such as payment or login.

Optional additional layers include user‑level, API‑key, and region‑level limits.

Configuration storm mitigation

A configuration storm occurs when many resources change simultaneously, causing repeated recomputation, snapshot generation, and xDS push overload. Typical triggers are bulk HTTPRoute creation, massive rolling updates, simultaneous certificate changes, and GitOps batch commits.

Mitigation principles:

Debounce control‑plane processing (e.g., TRANSLATOR_DRAIN_MILLISECONDS, TRANSLATOR_MAX_SNAPSHOTS).

Batch business changes instead of submitting hundreds of routes at once.

Split routes and policies by domain to limit impact scope.

Separate capacity models for high‑frequency Endpoint churn versus relatively stable routes.

Instrument xDS push latency, reject count, and snapshot size as core observability metrics.

controller:
  env:
    - name: RELATION_INFORMER_RESYNC_INTERVAL
      value: "30s"
    - name: TRANSLATOR_DRAIN_MILLISECONDS
      value: "100"
    - name: TRANSLATOR_MAX_SNAPSHOTS
      value: "1000"

Observability stack

Four metric layers are essential:

Traffic layer : request count, QPS, status‑code distribution, latency percentiles, downstream connections.

Governance layer : rate‑limit hits, retry count, circuit‑breaker trips, mirroring volume, auth failures.

Control‑plane layer : configuration translation latency, xDS push latency, snapshot count, config reject count, resource status errors.

Resource layer : total routes, listeners, clusters, endpoints, certificate expiry.

Sample alerts:

# 5%+ 5xx error rate
sum(rate(envoy_http_downstream_rq_xx{envoy_response_code_class="5"}[5m])) /
sum(rate(envoy_http_downstream_rq_total[5m])) > 0.02

# 99th‑percentile translation duration > 2 s
histogram_quantile(0.99, rate(kgateway_translation_duration_seconds_bucket[5m])) > 2

# Connection‑pool utilization > 80 %
envoy_cluster_upstream_cx_active / envoy_cluster_upstream_cx_max > 0.8

Multi‑cluster evolution

Three maturity stages:

Stage 1 : Single‑cluster single entry – quick to adopt but creates a single point of failure and governance bottleneck.

Stage 2 : Single‑cluster multiple gateways – isolates domains, enables independent scaling, and reduces fault domains.

Stage 3 : Multi‑cluster, multi‑region – provides cross‑region disaster recovery, proximity routing, and independent upgrades, at the cost of higher configuration‑consistency effort.

Cross‑cluster failover is expressed via Backend and TrafficPolicy resources that declare primary and secondary upstreams with priority and health‑check policies:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: payment-dr
  namespace: payment
spec:
  type: Static
  endpoints:
    - addr: payment-dr.example.internal
      port: 8080
  tls:
    mode: STRICT
    sni: payment-dr.example.internal
---
apiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata:
  name: payment-failover
  namespace: payment
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: payment-api
  failover:
    - from: payment-service
      to: payment-dr
      priority: 1

Migration strategy

Inventory existing domains, paths, certificates, upstream services, and change frequency.

Define ownership boundaries and design Gateway / Route ownership.

Migrate low‑risk, low‑coupling entry points first and validate platform templates.

Introduce gray release and mirroring at the gateway layer before cutting over.

Gradually template authentication, rate limiting, timeout, and retry policies.

Finally migrate high‑value core flows and perform certificate and disaster‑recovery rehearsals.

Production checklist

Pre‑release

Gateway / Route / Policy responsibility matrix completed.

Namespace authorizations and admission checks in place.

Gray release, rollback, and mirroring paths verified.

Certificate rotation process tested.

High‑peak capacity load‑testing performed.

Alerting for entry error rate, xDS latency, and connection‑pool usage configured.

During release

Monitor P95 / P99 latency drift.

Watch 4xx / 5xx and reset metrics.

Observe Route status and xDS reject counts.

Track upstream connection‑pool usage and retry counts.

Validate gray‑release hit rules match expectations.

Post‑release

Compare traffic, latency, and error rates against previous version.

Check metric label cardinality for explosion.

Verify trace and log sampling rates.

Document the change as a reusable template.

TLS, certificate rotation and long‑connection risks

Certificate rotation triggers Secret changes, TLS context updates on Listeners, and filter‑chain refreshes. If performed abruptly, long‑lived connections may be reset and TLS handshake failures spike.

Recommended “certificate‑drain” process:

Observe current active‑connection ratio before rotation.

Execute rotation during low‑traffic windows.

Gracefully drain listeners so new connections use the new certificate while existing connections finish.

Monitor handshake failure rate, downstream resets, and HTTP 5xx.

If needed, switch listeners one by one.

Route scaling considerations

Performance is not solely a function of route count. Critical factors are host aggregation, prefix distribution, regex usage, number of Listeners, and VirtualHost fragmentation. A well‑structured 5 000‑route set can be faster than a poorly organized 500‑route set.

Observability recommendations

Four metric layers (traffic, governance, control‑plane, resource) must be instrumented together to answer:

Why did a request fail?

Was it a rule miss, policy block, upstream timeout, or certificate issue?

Is the problem isolated to a single business flow or the entire gateway?

Was the root cause a configuration change, traffic surge, or backend degradation?

Essential alerts (error rate, translation latency, connection‑pool saturation) should be complemented with certificate expiry, xDS reject, route‑status, and rate‑limit hit alerts.

Key takeaways

Gateway API reshapes entry governance into clear resource layers.

Control‑plane / data‑plane separation decouples configuration churn from request processing.

Semantic compilation (Watcher → Translation Engine → xDS Snapshot → ADS) enables high‑frequency updates without proxy reloads.

Production stability requires explicit capacity models (throughput, connections, configuration), tiered rate limiting, and robust observability.

Configuration storms are mitigated by debouncing, batching, and domain‑level route splitting.

TLS rotation must be treated as a controlled deployment, not a simple Secret update.

Multi‑cluster deployments add disaster‑recovery and proximity benefits but increase configuration consistency complexity.

Migration should be domain‑by‑domain, preserving governance boundaries rather than a one‑click Ingress‑to‑Gateway conversion.