Why We Switched from Ingress‑Nginx to Apache APISIX Ingress: Lessons Learned

Project Background

In recent years, the market has seen a growing variety of products that can serve as Ingress controllers, expanding the selection range.

These products can be roughly divided into two categories: one like K8s Ingress and Apache APISIX Ingress, which are based on traditional proxies such as Nginx or OpenResty and use a K8s client written in Go for the controller; the other emerging solutions implement both proxy and controller functions directly in Go, such as Traefik.

YouPai Cloud originally used Ingress‑Nginx for most of its services.

The lower layer is the data plane OpenResty. The upper controller watches APIServer resource changes and generates

nginx.conf

files, then reloads OpenResty. When a POD IP changes, it can be passed to OpenResty Lua code via an HTTP interface to replace upstream nodes.

Ingress‑Nginx’s extensibility mainly relies on Annotations; its configuration syntax and routing rules are simple, and Lua plugins can be added for custom functionality.

However, Ingress‑Nginx has several drawbacks:

Complex component dependencies make plugin development hard and reduce portability.

Weak semantic capabilities.

Any Ingress modification requires a reload, which is unfriendly to long‑lived connections.

These issues prompted us to look for alternative container gateways.

Research Phase

We evaluated Traefik and Apache APISIX Ingress as replacements.

Traefik offers a cloud‑native model with extremely simple configuration, distributed discovery, and good Go ecosystem support, but it suffers from weak extensibility, no reload support, and lower performance compared to Nginx.

Consequently, we did not choose Traefik.

Why Choose Apache APISIX Ingress

Internal Reasons

Our data centers already run Apache APISIX gateway clusters that replaced Kong, and we have built internal plugins (e.g., permission checks, precise rate limiting) on top of its plugin system.

Operational Efficiency

Many of our Kubernetes clusters still use Ingress‑Nginx, and we have custom plugins there. Since APISIX and Nginx share many capabilities, consolidating on APISIX reduces duplicate development and operations.

Adjustments After Adopting Apache APISIX Ingress

Apache APISIX Ingress is still in early testing at YouPai Cloud; we are currently trialing it in new clusters.

Architecture Changes

After switching, the architecture looks like the following:

The data plane is now an Apache APISIX cluster. The controller watches APIServer changes and distributes configuration to all APISIX nodes via etcd.

Unlike Ingress‑Nginx, APISIX supports dynamic routing without reloading; a single Location forwards traffic and Lua code selects routes, simplifying deployment. In contrast, Nginx requires complex

nginx.conf

changes and reloads for every update.

Apache APISIX Ingress Controller

The controller relies on APISIX’s dynamic routing, monitoring APIServer resources, performing data conversion, validation, diff calculation, and applying changes via the APISIX Admin API. High availability is achieved through Kubernetes leader‑election without extra components.

Declarative Configuration

Two declarative options are supported: Ingress Resource and CRD Resource. The latter offers stronger semantics and is the choice for YouPai Cloud, allowing any APISIX capability to be expressed.

Benefits Achieved

Improved Logging Efficiency

Unified APISIX clusters across data centers simplify log processing. We use the Kafka‑Logger plugin with custom log formats.

Enhanced Monitoring and Health Checks

APISIX integrates with Prometheus (we use Prometheus) for metrics. Health checks are performed by deploying a service with multiple PODs and verifying the full chain to confirm APISIX health.

Practical Solutions Using Apache APISIX Ingress

Scenario 1: K8s Application Updates

Prefer rolling updates to keep most PODs available while handling load stability.

Ensure PODs pass internal health checks before receiving traffic.

Maintain high availability of APISIX upstream endpoints.

We observed occasional transmission delays during POD readiness propagation, sometimes extending beyond the typical 1‑second sync.

Solution: after a batch of PODs becomes Ready, wait a few seconds before updating the next batch to ensure most endpoints remain functional.

Scenario 2: Upstream Health Checks

Due to APIServer’s state‑driven design, synchronization delays can cause 502 errors. Direct upstream health checks are costly; instead, we perform passive TCP health checks at the gateway layer to detect connection timeouts and trigger fallback handling.

Scenario 3: mTLS Connection to etcd

APISIX clusters use one‑way TLS by default. When connecting to a Kubernetes etcd cluster that requires mutual TLS, APISIX’s HTTP‑to‑gRPC‑gateway adds an extra verification step, requiring the server certificate to be usable for both client and server authentication.

Additionally, OpenResty does not support mTLS natively; to use mTLS with upstream services or etcd, a patched OpenResty built with the apisix‑nginx‑module is needed (available via apisix‑build‑tools).

Future Expectations

Although APISIX Ingress is currently in testing, we anticipate that after further iterations and architectural migration, it will become the unified gateway for all of YouPai Cloud’s container workloads.

About Apache APISIX

Apache APISIX is a dynamic, real‑time, high‑performance open‑source API gateway offering load balancing, dynamic upstreams, gray releases, circuit breaking, authentication, observability, and more.

It helps enterprises securely handle API and microservice traffic, supporting gateway, Kubernetes Ingress, and service mesh scenarios.

Apache APISIX GitHub: https://github.com/apache/apisix

Apache APISIX website: https://apisix.apache.org/

Apache APISIX documentation: https://apisix.apache.org/zh/docs/apisix/getting-started