How We Unified a Complex Multi‑Gateway Architecture with Higress and Istio CRD

Background

The platform operated several gateways—OpenResty, Istio IngressGateway, Spring Cloud Gateway, Kong, Node‑Express, and APISIX—resulting in over 20 w configuration entries for roughly 500 services and 400 sub‑domains. Diverse routing types (service, build‑class, return/rewrite) and legacy rules caused high operational pressure and frequent conflicts.

Business Demands

Development teams required extensible capabilities such as authentication, security controls, and request‑body modifications, prompting a unified gateway solution.

Project Goals

Govern gateway configuration to reduce complexity and risk.

Unify the gateway technology stack to improve operational efficiency and support common scenarios (forwarding, rewriting, protocol conversion, plugin extensions, future security and traffic‑coloring features).

Solution Overview

Configuration Governance

Standardization : Defined domain‑usage and rule‑configuration standards and integrated them into the change‑management workflow.

Layered Rules : Split rules into a domain layer (managed by ops) and a service layer (managed by developers).

Application‑Centric Management : Linked service‑layer rules to application metadata so their lifecycle matches the application’s.

Technology‑Stack Unification

After evaluating community activity and feature sets, the team selected three candidates—APISIX, Higress, and Istio IngressGateway. Higress was chosen because it wraps Istio’s capabilities, offers stronger plugin support, and simplifies virtual‑service configuration.

Why Istio CRD Over Ingress/Gateway API

Istio’s VirtualService resource allows a single object to express many hosts, keeping configurations concise and readable, whereas Kubernetes Ingress would require one resource per host and extensive annotations. Gateway API was still immature for the required features.

Migration Challenges and Resolutions

Domain aggregation (enableSRDS) caused configuration explosion; disabling it fixed the issue.

Initial slow rule propagation (minutes) was optimized by the Higress community to seconds.

Istio versions < 1.22 had rule‑conflict activation bugs; the community patched them in 1.22 and Higress 1.4.1.

Higress initially used more memory than Istio; after metric pruning, usage became comparable.

Gateway Layering Design

A custom AppVirtualService (AVS) CRD was introduced to hold service‑layer routing without domain information. Example:

apiVersion: arch.cai-inc.com/v1
kind: AppVirtualService
metadata:
  labels:
    app: web-a-front
    vsLayer: app
  name: web-a-front
  namespace: test
spec:
  http:
  - match:
    - uri:
        prefix: /a-index
    route:
    - destination:
        host: web-a-front
        port:
          number: 80

The AVS includes a PriorityClass (high, medium, low) to control rule precedence, mitigating the previous “first‑come” ordering.

Automation Tooling

A conversion tool extracts existing VirtualService objects, splits them into domain‑layer VirtualService and service‑layer AVS, and generates the layered configuration. Routes are categorized as domain‑default, service (conflicting or non‑conflicting), and return/rewrite. Non‑conflicting service routes (≈90 %) become AVS resources, enabling developers to manage them directly.

Data Consistency Verification

Original traffic logs are replayed against a test environment with the layered rules. The replayed logs are compared to the production logs; any request (same domain + URL) that forwards to a different upstream indicates a regression.

Key steps:

Generate a unique key using md5(domain+url), normalizing variable IDs via regex.

Record domain, URL, upstream, request ID, etc., to a file for replay.

Replay using HEAD requests to avoid side effects.

Gradual Rollout

New gateway instances receive the layered rules. Traffic is shifted incrementally (1 % → 5 % → 10 % → 20 % → 50 % → 100 %) to the new instance, ensuring stability.

Initial Outcomes

Developers experienced virtually no impact during the migration.

Rule governance goals were met: rules are now organized by domain and application layers.

Technology stack unified: Higress replaced the container gateway, Kong and Spring Cloud Gateway were retired, and APISIX usage was reduced, saving resources and maintenance effort.

Future work includes extending Higress with open‑platform capabilities (authentication, HTTP‑to‑Dubbo conversion) and further consolidating the gateway ecosystem.

Illustrations