Why Sealos Chose Higress: A Deep Dive into Cloud‑Native Gateway Evaluation

Background and Requirements

Sealos public cloud has grown to over 87,000 registered users, each creating applications that need individual access endpoints, resulting in hundreds of thousands of Ingress entries. The platform must support massive Ingress scalability, strict multi‑tenant isolation, traffic control, and robust security against external attacks.

Why Nginx Ingress Was Dropped

Reloading on every Ingress change caused brief connection loss, which became frequent at scale.

Long‑living connections were unstable due to frequent changes.

Performance was poor: slow activation, high resource consumption, and occasional OOM crashes.

These issues led to the decision to eliminate Nginx‑based gateways.

Envoy‑Based Gateways Show Superior Performance

Benchmarks demonstrated that Envoy implementations consume far less control‑plane and data‑plane resources, handling massive routing tables without degrading performance.

APISIX Evaluation

APISIX solved many Nginx reload problems, but its Ingress Controller suffered from instability, controller crashes, and OOM incidents, making it unsuitable for Sealos’s production needs.

Cilium Gateway Limitations

Cilium Gateway only supports Load‑Balancer mode, tying the solution to cloud provider LB services and failing to meet private‑cloud scenarios. Additionally, route activation could take minutes, far exceeding the desired sub‑5‑second window.

Envoy Gateway Early‑Stage Issues

Although conceptually aligned with Sealos’s preference for Envoy, Envoy Gateway is still immature, exhibiting bugs such as OOM, non‑functional path policies, and missing features in merge‑gateway mode.

Gateway Standard Design Gaps

The current Kubernetes Gateway API mixes admin‑level fields (e.g., listener ports) with tenant‑level configurations, lacking clear permission separation. A more elegant design would move tenant‑specific fields into HTTPRoute or a dedicated CRD, allowing distinct admin and user privileges.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
  - name: http
    port: 80
    protocol: HTTP
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: example-com

Final Decision: Higress Wins

After a systematic elimination process, Higress emerged as the only gateway meeting Sealos’s production stability criteria. Key achievements include:

Ingress activation time reduced from >2 minutes to ~3 seconds via incremental configuration loading.

Resolved controller OOM issues and eliminated high‑resource consumption.

Mitigated occasional timeout problems by disabling the onDemandRDS feature.

Outstanding performance under massive routing and high concurrency, with minimal resource usage (see charts).

Full compatibility with existing Nginx Ingress annotations, enabling near‑zero migration effort.

Community support from the Higress team quickly addressed remaining bugs, reinforcing its reliability.

Community Feedback and Future Improvements

Better alignment with the evolving Gateway standard (full v1 compatibility).

Expose advanced security and circuit‑breaking capabilities, potentially as paid extensions.

Adopt a plugin architecture for peripheral features, keeping the core lightweight.

Conclusion

Gateways are critical for cloud platforms and applications. Sealos’s rigorous evaluation highlights the challenges of multi‑tenant, high‑scale environments and provides a reference for others facing similar decisions. The team will continue to contribute to open‑source gateway projects and collaborate with upstream communities.