How Alibaba Cloud AI Gateway Ensures High Availability for LLM Services

Problem Overview

LLM services have large model sizes, leading to long deployment and restart times. Overload can cause minutes‑long outages, severely impacting availability.

AI Gateway High‑Availability Features

Alibaba Cloud AI Gateway provides multi‑source LLM proxy with traffic governance, passive health checks, first‑packet timeout, and fallback routing to protect services during spikes.

Model and Resource

Example model: DeepSeek‑R1‑Distill‑Qwen‑7B using resource ml.gu7i.c8m30.1-gu30 (24 GB GPU). Under load GPU utilization reaches 99 % and first‑packet response time (RT) grows with request count.

Fallback Mechanism

Create a gateway instance and add an AI service.

Select model provider (e.g., PAI‑EAS) and the specific model.

Enable the fallback option and choose an alternative service such as Alibaba Cloud Baichuan.

After creation, the LLM API can be debugged directly from the gateway to verify request flow.

Passive Health Check & First‑Packet Timeout

First‑packet timeout : If the initial response exceeds a threshold (e.g., 200 ms), the request fails fast, prompting a retry.

Passive health check : When the failure rate exceeds a threshold (e.g., 50 %), the node is marked unhealthy and ejected for a base time (e.g., 30 s). The ejection interval grows with repeated failures and shrinks on recovery.

Typical configuration:

Failure rate threshold: 50 %

Check interval: 1 s

Base ejection time: 30 s

First‑packet timeout: 200 ms

End‑to‑End Flow

Create the AI gateway service and enable passive health check with the parameters above.

Configure the LLM API, setting the first‑packet timeout to 200 ms.

Enable fallback to Baichuan.

During a traffic surge, the gateway monitors GPU usage and first‑packet latency. If latency exceeds the timeout, requests fail fast; the failure rate quickly reaches the 50 % threshold, causing the primary PAI‑EAS node to be ejected. Traffic is then routed to Baichuan, ensuring continuous service. When the primary node recovers, it is gradually reintegrated.

Key Benefits

Fast failure detection via first‑packet timeout reduces user‑perceived latency.

Passive health checks automatically isolate overloaded nodes.

Fallback routing provides seamless continuity with a backup LLM.

This combination of proactive health monitoring, timeout control, and fallback routing enables reliable LLM service operation under bursty traffic conditions.