Which Model Should Handle Your Request? Alibaba Cloud AI Gateway Intelligent Routing Goes Live

When applications need to serve diverse tasks—such as lightweight Q&A, summarization, complex reasoning, code generation, or Agent tool calls—their underlying models differ in ability, price, latency, and availability, making model selection increasingly difficult.

Common approaches either funnel all traffic to a flagship model (ensuring correctness but inflating cost and rate‑limit pressure), embed static selection logic in code (requiring redeployment whenever a model changes), or re‑evaluate on every request (risking context loss and added latency in Agent chains). The AI Gateway’s intelligent routing aims to eliminate these pain points by centralising model selection within the gateway.

Routing workflow (six steps) :

Identify whether the request is an independent call, a normal user message, or a continuation of a tool/model state.

Filter out candidates that fail to meet mandatory criteria such as protocol, modality, tool capability, context window, authorization, or current availability.

Interpret the current task, its difficulty, and the indispensable capabilities it requires.

Using ModelCard metadata and platform‑provided capability references, build a candidate set that can fulfil the task.

Apply the caller‑specified Profile (e.g., auto/cheap, auto/fast, auto/quality, auto/balanced) to rank the candidates.

Generate a unique routing decision and forward the request to the chosen model.

The system distinguishes two important boundaries:

Task advisor vs. gateway policy : The advisor only describes required capabilities; the gateway enforces hard constraints (ability, cost, health) and may fall back to a locally‑scored model when the advisor is unavailable.

Routing vs. fallback : Intelligent routing selects a single primary model for the request; retry and fallback logic remain separate and are not mixed into the routing chain.

For independent requests , the gateway treats each call as a new decision unit, allowing a fresh model choice each time. In multi‑turn conversations , the gateway only switches models when it confidently detects a new task; otherwise it conservatively re‑uses the previously executed model to preserve context. For Agent tool chains , the gateway decides the model once at task start and keeps it consistent throughout the chain, avoiding mid‑task model swaps that would break cache affinity.

Model candidates are described by a ModelCard containing structured facts—supported protocols, input/output modalities, tool and reasoning capabilities, context capacity, and a Credit score used for routing preferences (not billing). Users can select up to five custom candidates or enable built‑in platform tiers (economy, standard, flagship) and mix them in a single decision.

The Credit reflects the user’s cost or internal pricing model; it is independent of the actual service‑level billing performed by the underlying model provider.

Integration requires no new SDK. In the console, enable “Intelligent Routing” for a Model API, configure custom candidates or platform tiers, and publish. Calls simply set the model field to an explicit profile, for example:

{
  "model": "auto/balanced",
  "messages": [
    {"role": "user", "content": "请总结这份材料并列出三个行动项"}
  ]
}

If a concrete model name is supplied, the gateway bypasses routing and preserves the original behaviour.

Security-wise, the Model API enforces consumer authentication, isolates tenant state, and stores only bounded task summaries—not full prompts or credentials—ensuring sensitive data does not appear in logs.

Key boundaries include:

Only explicit auto/<profile> requests are handled by intelligent routing.

Each request yields a single primary model; no automatic fallback chain is generated.

Profiles express optimisation preferences but do not guarantee the absolute cheapest, fastest, or strongest model on every call.

Continuity relies on observable protocol evidence; when context is compressed or causal information is lost, the system degrades conservatively rather than guessing across sessions.

In summary, the AI Gateway’s intelligent routing consolidates model‑selection logic into a clear entry point, giving independent tasks a choice space, preserving stability in multi‑turn sessions, and preventing redundant decisions in Agent tool chains while ensuring the chosen model meets capability requirements before optimising for cost, speed, or suitability.