How SOFA AI Gateway Transforms Cloud‑Native AI Service Management

Background

Traditional gateways handle traffic governance, routing, protocol conversion and security for services. AI workloads shift the focus from services to models and agents, requiring new interaction patterns, resource management and risk models.

Why a Dedicated AI Gateway?

AI scenarios such as service models, agents, AI applications and Model‑Calling‑Protocol (MCP) demand capabilities that generic API gateways cannot provide. An AI‑focused gateway adds intelligent routing, unified model access, semantic caching, content security, MCP proxy and fine‑grained model rate‑limiting.

SOFA AI Gateway Overview

Ant Group’s SofaStack builds the SOFA AI Gateway (also called SOFA Higress) on the open‑source Higress kernel. The gateway is positioned as a high‑performance, stable and secure entry point for enterprise AI applications.

Core AI Business Scenarios

Agent Proxy – Provides a unified ingress/egress for agent traffic, security protection, flow control and a Tools Hub that converts existing REST APIs into agent‑callable functions, including REST‑to‑MCP conversion.

Model Proxy – Offers model inference with semantic cache, content security, unified access and precise model rate‑limiting.

MCP Market Service – Supplies a financial‑focused MCP marketplace that delivers professional data and services to accelerate agent development.

Implementation Details

3.1 Technical Selection

The gateway adopts Higress because of its active open‑source community, extensibility and compatibility with future multi‑gateway integration. Existing API, data and inter‑gateway capabilities are migrated to Higress.

3.2 Agent Entry‑Exit Gateway

Defines a unified entry for agent traffic, ensuring security, stability and seamless integration with external systems via tool and MCP management.

3.3 Inference Gateway – Model Intelligent Routing

Model services differ from traditional services: they have high latency, GPU‑heavy resource consumption and variable processing times. Simple load‑balancing (round‑robin, least connections) is ineffective. The gateway implements dynamic routing based on real‑time instance load, KV‑cache status and queue length, and supports model registration, deregistration and lifecycle management.

3.4 MCP Market

Professional MCPs provide precise, real‑time capabilities and authoritative data, especially for finance. By packaging financial analysis, diagnostics and market insights as MCPs, the gateway offers a SaaS‑style “Lego” market for agents.

Intelligent Routing Architecture

SOFA AI Gateway implements routing logic as Higress plugins, avoiding modifications to the data‑plane core. Plugins can invoke external EPP services via the ext‑proc protocol or standard HTTP, enabling custom routing decisions without rebuilding the gateway.

Challenges and Future Work

Entity Extraction Accuracy – Natural‑language queries often contain aliases or informal terms, leading to ambiguous financial entity resolution. A “slot‑filling” capability is planned to refine entity mapping.

MCP Context Explosion – As the number of MCPs grows, request contexts can become excessively large, degrading model performance. An intelligent MCP routing mechanism will select only the necessary MCPs per request.

Future work includes implementing slot‑filling, intelligent MCP routing and eventually integrating native Gateway API Inference Extension support into the Higress data plane.

Reference URL for the MCP marketplace: https://mcp.sofa.antdigital.com/mcp/home