How Tengine-Ingress Boosts Cloud‑Native Traffic with Zero‑Downtime Updates

Introduction

Unified Access, the 7‑layer application traffic entry gateway of Alibaba Group, relies on Tengine as a high‑performance HTTP and reverse‑proxy web server, handling tens of millions of QPS daily and supporting major events like Double‑11. However, limitations such as static domain and certificate updates, weak configurability, and lack of gray‑release capabilities have long been pain points.

Terminology

AServer‑Ingress: A new cloud‑native ingress‑standard application that provides a stable 7‑layer traffic entry gateway for various Alibaba business units.

Tengine‑Ingress: The technical foundation of AServer‑Ingress, built on an optimized Kubernetes/ingress‑nginx combined with Tengine‑Proxy.

K8s: Kubernetes, a distributed system for deploying, scaling, and managing containerized applications.

ASI: Alibaba Cloud Serverless Infrastructure, offering managed and semi‑managed K8s clusters.

Ingress: Native K8s resource for HTTP(S) L7 load balancing, TLS termination, and routing.

Annotations: Ingress annotations used to extend L7 load‑balancing, TLS, and routing configurations.

etcd: Distributed key‑value store for K8s configuration data.

API server: Core K8s component that validates and updates resources in etcd.

CRD: CustomResourceDefinition, allowing user‑defined resources in the API server.

MD5: Message‑Digest Algorithm for generating 128‑bit hash values.

Architecture

Public HTTPS requests first reach the Layer‑4 SLB, which forwards traffic to the Layer‑7 AServer‑Ingress gateway. AServer‑Ingress performs local TLS termination using dynamic ingress domains and secrets, then routes HTTP requests to backend services based on unitized routing policies.

Tengine‑Ingress consists of a controller and a proxy. The controller watches ingress and secret resources, validates them, and writes dynamic configurations to shared memory. The proxy reads this shared memory and forwards external traffic to K8s services.

Cloud‑Native Features

Tengine‑Ingress fully complies with the K8s ingress standard and adds several enhancements:

Dynamic, loss‑less domain and certificate updates without Tengine reload.

Dual‑certificate support (ECC and RSA) and extensibility to national‑cipher suites.

Gray‑release annotations for header, regex, cookie, and weight‑based traffic splitting.

Per‑domain observability (QPS, success rate, latency, payload size).

Advanced HTTP(S) configuration via ingress annotations.

tls:
  - hosts:
    - {host1}
    secretName: {secret name1}
  - hosts:
    - {host1}
    secretName: {secret name2}

High Availability

All configuration changes are gray‑released, monitored, and rollback‑able. Ingress domain updates are applied batch‑by‑batch using custom annotations (e.g., nginx.ingress.kubernetes.io/ingress-rollout, ...-current-revision, ...-update-revision, ...-rollout-index-id). This limits the impact of faulty configurations to a subset of pods, allowing immediate termination or rollback based on success metrics.

Global Consistency Checks

Tengine‑Ingress computes a global MD5 checksum for all ingress resources. If the checksum matches the value stored in the custom resource IngressCheckSum, the configuration is considered globally consistent and is applied; otherwise, the system rejects the update and continues serving the previous stable configuration.

type IngressCheckSum struct {
  metav1.ObjectMeta
  Spec IngressCheckSumSpec
}

type IngressCheckSumSpec struct {
  Timestamp metav1.Time
  Checksum string
  Ids []string
}

Performance

Compared with the legacy Tengine gateway, Tengine‑Ingress performs TLS termination locally, eliminating remote Keyless service calls. In HTTPS 1.1 short‑connection tests, average response time improves by ~9 ms, CPU usage drops from 81.9 % to 67.3 % (≈20.7 % reduction), and memory consumption falls from 20.4 % to 8.8 %.

Current Status and Outlook

Tengine‑Ingress is deployed across all Alibaba online, pre‑release, VPN, and intranet scenarios, handling hundreds of thousands of domains with a 57 % reduction in memory usage and ~20 % performance improvement in short‑connection HTTPS traffic. Future plans include extending the gateway to support 4/7‑layer TLS, TCP, UDP, and gRPC routing, evolving toward a universal API gateway.

Official sites: https://tengine.taobao.org GitHub repositories: tengine‑ingress , tengine , xquic , libxudp