Scaling AI Agents with Semantic Tool Selection Using AgentScope Java and Higress

Background and Challenges

When AI agents are used in enterprise scenarios, developers attach many tools (weather, maps, database APIs, internal services). As the number of tools grows to hundreds or thousands, the traditional “expose all” model becomes unsustainable: prompts become excessively long, tool‑selection errors increase, response latency rises, and inference cost explodes.

Core Challenges of Large‑Scale Tool Management

Prompt bloat – Each tool requires a name, description and schema in the prompt, quickly exhausting the LLM context window.

Uncontrolled inference cost – Longer prompts raise token consumption, leading to exponential cost growth.

Reduced tool‑selection accuracy – Similar or irrelevant tools cause the model to mis‑choose.

Increased response latency – Processing long contexts slows inference.

Higher maintenance complexity – Manual filtering of visible tools limits scalability.

Security and stability risks – Accidental selection of unrelated tools can cause invalid calls or data leakage.

Semantic‑Driven Tool Selection

AgentScope Java integrates the Higress AI Gateway. The gateway performs semantic retrieval of tools based on the agent’s natural‑language intent and injects only the top‑K relevant tools at runtime, following the principle of least privilege and reducing context size.

Key Capabilities of the Higress Extension

Semantic‑driven tool selection: retrieve the most relevant tools from a natural‑language description.

Zero‑intrusion MCP client: standard Java client compatible with existing AgentScope ecosystems.

Enterprise‑grade observability and security: authentication and authorization via Alibaba Cloud AI Gateway.

Performance Evaluation

The retrieval uses a hybrid Weight algorithm that combines vector similarity with re‑ranking. Accuracy tests show a slight edge over pure vector search, while latency stays under 350 ms (≈30 ms overhead compared to vector‑only).

Quick Start

1. Add Dependency

<dependency>
    <groupId>io.agentscope</groupId>
    <artifactId>agentscope-extensions-higress</artifactId>
    <version>${agentscope.version}</version>
</dependency>

2. Enable Semantic Tool Search

// Build a client with semantic search enabled
HigressMcpClientWrapper higressClient = HigressMcpClientBuilder.create("higress")
    .streamableHttpEndpoint(HIGRESS_ENDPOINT)
    // .sseEndpoint(HIGRESS_ENDPOINT + "/sse") // optional SSE transport
    // .header("Authorization", "Bearer xxx") // optional auth header
    // .queryParam("queryKey", "queryValue") // optional query param
    .toolSearch("your agent description", 5) // retrieve top‑K relevant tools
    .buildAsync()
    .block();

Toolkit toolkit = new HigressToolkit();
toolkit.registerMcpClient(higressClient).block();

ReActAgent agent = ReActAgent.builder()
    .name("HigressAgent")
    .sysPrompt("You are a helpful assistant. Please answer questions concisely and accurately.")
    .model(DashScopeChatModel.builder()
        .apiKey(apiKey)
        .modelName("qwen-max")
        .stream(true)
        .enableThinking(false)
        .formatter(new DashScopeChatFormatter())
        .build())
    .toolkit(toolkit)
    .memory(new InMemoryMemory())
    .build();

Prerequisites

Create an Alibaba Cloud AI Gateway instance (paid or pay‑as‑you‑go). URL: https://common-buy.aliyun.com/?commodityCode=apigateway_aipost_public_cn

Register an MCP tool service in the AI Gateway. URL: https://help.aliyun.com/zh/api-gateway/ai-gateway/user-guide/gateway-managed-mcp-services

Enable semantic search in the MCP Management > Semantic Retrieval tab.

(Optional) Configure consumer authentication for added security.

References

Full example: https://github.com/agentscope-ai/agentscope-java/blob/main/agentscope-examples/quickstart/src/main/java/io/agentscope/examples/quickstart/HigressToolExample.java

AgentScope Java repository: https://github.com/agentscope-ai/agentscope-java

Higress repository: https://github.com/alibaba/higress