Seamlessly Connect Dify Apps to Multiple AI Models with the Higress AI Gateway

Problem

When building AI applications with Dify, developers often need to integrate multiple model providers. This creates challenges such as complex multi‑model management, lack of unified governance (rate limiting, circuit breaking, intelligent routing, context compression), and the need for fine‑grained authentication beyond simple API keys.

Higress AI Gateway Solution

Higress AI Gateway provides a unified Model API that proxies various model vendors and self‑hosted inference services. It adds production‑grade capabilities including high‑availability traffic governance, security, and observability. A dedicated Dify plugin enables Dify to call models through this gateway without additional code.

Plugin Features

Supports text generation, image generation, vector embedding, and text ranking.

Provides OpenAI‑compatible and Alibaba Cloud Bailei protocols.

Authentication methods: API Key and HMAC (AK/SK).

Governance: token rate limiting, fallback, circuit breaking, timeout handling.

Load balancing across multiple backend models.

Observability with unified logs, metrics, and tracing.

Security with consumer authentication and IP whitelist/blacklist.

Extensible via the Higress plugin ecosystem.

Supported Scenarios and Protocols

Text generation – OpenAI‑compatible protocol – API Key / HMAC.

Image generation – Alibaba Cloud Bailei protocol – API Key / HMAC.

Vector embedding – OpenAI‑compatible protocol – API Key / HMAC.

Text ranking – Alibaba Cloud Bailei protocol – API Key / HMAC.

Demo 1: Agent Intelligent Assistant

Create Model API routes in the Higress gateway for LLM, Embedding, and Rerank models.

Install the Higress model plugin in Dify and create corresponding models that point to the gateway routes.

Build a knowledge base using the Embedding and Rerank models provided by the gateway.

Create an Agent application in Dify, select the LLM model from the plugin, configure prompts, attach the knowledge base, and enable required tools (e.g., time check).

When queried, the Agent retrieves relevant documents from the knowledge base, generates a response via the LLM, and can invoke tools to produce detailed product recommendations.

Demo 2: Workflow Image Generation

Create an image‑generation Model API route in the Higress gateway.

Install the Higress plugin in Dify and configure a model that uses the gateway route.

Build a Workflow that accepts a textual description, passes it to the image‑generation model, receives an image URL, and formats the output as markdown for display.

The workflow demonstrates end‑to‑end image generation with the model accessed via the Higress gateway.

Governance, Load Balancing, Observability, and Security

Traffic protection : token throttling, model fallback, circuit breaking, timeout policies.

Load balancing : intelligent distribution across multiple backend model instances.

Observability : unified request logs, metric collection, and distributed tracing.

Security : consumer authentication, IP whitelist/blacklist.

Extensibility : additional AI capabilities can be added through the Higress plugin ecosystem.

Future Work

Add more scenarios and protocols to align with emerging model APIs.

Provide finer‑grained traffic‑policy configuration integrated with Higress features.

Source Code

The plugin is fully open‑source. Repository:

https://github.com/higress-group/higress-dify-plugin