97% Latency Drop: Higress vs Alibaba Cloud AI Gateway Serverless Enterprise Compared

Context: AI Workloads Shift Toward Large Payloads

AI applications are moving from single-turn text Q&A to long-context, multimodal, and agent-driven calls. A single request may carry entire conversation histories, retrieved documents, or multiple images, and may invoke different models and tools. Larger request bodies and longer call chains raise the bar for AI gateway processing efficiency and runtime stability.

Benchmark Setup

Two deployments were compared: Alibaba Cloud AI Gateway Serverless Enterprise Edition (managed) and open-source Higress self-hosted . Both used the same load-test source code and synthetic JSON payloads against a shared mock backend that added no artificial think time, eliminating real model inference latency. Tests fixed request sizes at 1 MB and 10 MB , then ramped concurrency from 10 to 100. Results reflect each deployment's own resource provisioning and entry points.

Results: 1 MB Requests

At 10–100 concurrency, Serverless Enterprise average gateway processing latency stayed in the 13–18 ms range, while open-source Higress latency climbed with concurrency. The maximum reduction reached 97.1% at 90 concurrency: Higress averaged 446.2 ms vs. Serverless Enterprise 12.7 ms . Even at the smallest request size tested, the gap widened as concurrency increased.

Results: 10 MB Requests

For 10 MB payloads, Serverless Enterprise latency was 82.7%–90.2% lower. Peak reduction of 90.2% occurred at 30 concurrency: Higress 1,688.1 ms vs. Serverless Enterprise 166.1 ms . At 100 concurrency, Serverless Enterprise averaged 384.3 ms versus Higress 2,527.4 ms , a 84.8% reduction saving roughly 2.14 seconds per request.

Technical Analysis: Why the Difference?

The article explains the performance gap by examining plugin processing paths:

Wasm linear memory in Higress can grow but cannot shrink; objects released by plugins may be reused internally but are not immediately returned to the host. If a plugin holds the request body while waiting for the model response, memory occupancy extends.

Under sustained concurrency, unreleased data accumulates with new allocations, keeping memory pressure high and potentially degrading throughput and queueing latency.

Multiple Wasm plugins processing the same large payload can cause repeated copies and allocations, amplifying overhead as payload size and concurrency grow.

Serverless Enterprise uses native Envoy plugins optimized for AI request flows, reducing redundant data copies and unnecessary memory retention. These optimizations are continuously maintained by the platform.

Operational Comparison: Self-Hosted vs. Managed

Self-hosted Higress : Costs come from compute, network, and redundant replicas; capacity must be reserved for traffic peaks. Ongoing effort includes image maintenance, plugin compatibility checks, upgrade regression testing, and incident troubleshooting.

Serverless Enterprise : Platform manages underlying capacity and nodes. Teams configure models, routes, and policies. Pricing: instance fee 0.245 CNY/hour plus pay-per-use for calls, data processing, and public egress. Offers 99.95% SLA with platform-managed runtime guarantees and version maintenance, reducing infrastructure failure handling and service interruption risk.

Product Tiers

Serverless Standard : Free tier for trial, dev/test, low traffic; no instance fee, pay-per-use.

Serverless Enterprise : Production-grade, dedicated ingress, custom domain, TLS config; instance fee + usage.

Dedicated Instance : For stricter isolation, performance specs, and advanced governance.

References

AI Gateway Types and Version Capabilities –

https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/ai-gateway-capacity-description

Serverless New Version Billing –

https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/overview-of-billing-during-the-serverless-public-preview-of-the

AI Gateway Engine Version Description –

https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/engine-version-description