Spring AI 2.0 Tool Search: Stop Flooding LLMs with Dozens of Tools

When the author first built a customer-service AI Agent with Spring AI, three to four tools (order query, inventory, logistics) worked well. The model could chain calls without explicit orchestration code.

As the business grew, the Agent accumulated over 30 tools across orders, products, inventory, members, coupons, logistics, and after-sales. Although the code still ran, the model became less accurate: for a simple question like "Where is order 20260907001?" it had to choose from dozens of similarly named tools (getOrder, getOrderDetail, getOrderStatus, getRefundOrder, getRefundStatus, etc.) every request.

Previous Workaround: Manual Agent Splitting

The author initially split the monolithic Agent into domain-specific Agents (Order Agent, Refund Agent, Logistics Agent) with a router Agent on top. This worked but introduced a new routing system just to manage tools.

Spring AI 2.0 Tool Search

Upgrading to Spring AI 2.0.x (with Spring Boot 4) enabled the spring-ai-starter-tool-search-advisor. The flow changes from:

User question → 30 tools sent to model → Model chooses tool → Execute

to:

User question → Model gets toolSearchTool → Searches needed capabilities → Only relevant tools added to context → Model calls actual business tools

The model initially sees only a search tool; the 30+ business tools are indexed (Lucene by default) and retrieved on demand.

Configuration

Maven dependencies:

org.springframework.ai:spring-ai-bom:2.0.1 (pom, import)
org.springframework.ai:spring-ai-starter-model-openai
org.springframework.ai:spring-ai-starter-tool-search-advisor

YAML configuration:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
    chat:
      client:
        tool-search-advisor:
          enabled: true
          tool-index-type: lucene
          max-results: 5

Enabling spring.ai.chat.client.tool-search-advisor.enabled=true swaps the default Tool Calling Advisor for ToolSearchToolCallingAdvisor. Lucene was chosen because tool descriptions already contained strong domain keywords (order, inventory, logistics, refund), making embeddings unnecessary.

Tool Definitions Unchanged

Business tools remain standard Spring AI @Tool beans. Example OrderTools:

@Component
public class OrderTools {
    private final OrderService orderService;
    public OrderTools(OrderService orderService) { this.orderService = orderService; }
    @Tool(description = """
        Query basic order information.
        Use when user asks about order status, payment status, shipping status,
        order time, or order amount.
        """)
    public OrderInfo getOrder(
        @ToolParam(description = "Order number") String orderNo) {
        return orderService.getOrder(orderNo);
    }
    @Tool(description = """
        Query order item details.
        Use when user needs to know which products, SKUs, quantities, prices in an order.
        """)
    public List getOrderDetail(
        @ToolParam(description = "Order number") String orderNo) {
        return orderService.getOrderItems(orderNo);
    }
}

LogisticsTools

follows the same pattern.

Critical: Rich Tool Descriptions

Short descriptions like @Tool(description = "Query logistics") fail with Tool Search because the search tool must understand when to use each tool. The author rewrote descriptions to include trigger scenarios:

Query current logistics info for an order.
Applicable when user asks where the package is, whether it has been picked up,
transport progress, or estimated delivery status.

This mirrors building an Elasticsearch index: poor source data cannot be fixed by a better search algorithm.

ChatClient Setup

All tools are still registered via defaultTools() so Spring knows they exist, but Tool Search prevents sending all definitions to the model each turn.

@Configuration
public class AiConfig {
    @Bean
    ChatClient customerServiceChatClient(ChatClient.Builder builder,
        OrderTools orderTools, LogisticsTools logisticsTools,
        RefundTools refundTools, ProductTools productTools,
        InventoryTools inventoryTools, MemberTools memberTools) {
        return builder
            .defaultSystem("""
                You are a mall customer-service assistant.
                Call system-provided business tools to get real data; do not fabricate.
                If existing tools are insufficient, first search for usable tools.
                For write operations (cancel, refund, address change), clearly state the action before executing.
                """)
            .defaultTools(orderTools, logisticsTools, refundTools,
                          productTools, inventoryTools, memberTools)
            .build();
    }
}

A conversationId (session ID) is passed per request because Tool Search indexes tools per session.

Observed Behavior

For "Order 20260907001 why not received?" the model now first searches with terms like "order shipping logistics delivery", receives only order and logistics tools, then calls getOrder("20260907001") → gets tracking number → calls getLogisticsTrace("SF123456789") → replies with real data.

Write-Operation Safety

Direct write tools (e.g., cancelOrder) are risky. The author wraps them in a confirmation layer:

@Tool(description = """
    Create a cancel-order request.
    Use when user explicitly asks to cancel an order.
    This tool only creates a pending action; it does NOT cancel immediately.
    """)
public PendingAction prepareCancelOrder(
    @ToolParam(description = "Order number") String orderNo) {
    return actionService.create(ActionType.CANCEL_ORDER, orderNo);
}

Returns {actionId, type, orderNo, status: WAIT_CONFIRM}. User confirms, then Java service verifies permissions, order state, idempotency, and executes the real cancellation inside a transaction.

@Transactional
public void confirmAction(Long userId, String actionId) {
    PendingAction action = actionRepository.findByActionId(actionId).orElseThrow();
    if (!Objects.equals(action.userId(), userId)) throw new AccessDeniedException("No permission");
    if (action.status() != WAIT_CONFIRM) throw new IllegalStateException("Already processed");
    switch (action.type()) {
        case CANCEL_ORDER -> orderService.cancel(userId, action.bizId());
        case REFUND -> refundService.createRefund(userId, action.bizId());
        default -> throw new UnsupportedOperationException();
    }
    actionRepository.markCompleted(actionId);
}

Principle: AI decides what to do; Java backend decides whether and how safely to do it.

Architectural Shift

The author notes that Java Agent development is converging with traditional backend concerns: permissions, transactions, auditing, timeouts, retries, idempotency, circuit breaking — previously protecting Controllers, now also protecting Agents.

Tool Search doesn't make the model smarter; it solves the engineering problem of managing a growing set of business capabilities exposed to an Agent. With 5 tools, no need. With 20–30+ tools (or future MCP integrations), stuffing all definitions into every context becomes unsustainable.

After the change, the author no longer splits Agents by domain. Tools stay organized by Java domain services ( OrderTools, RefundTools, LogisticsTools, etc.), and Tool Search filters the relevant subset per request.

Next Step: Tool Permissions

Different roles (customer service, finance, operations) should see different tool subsets. The envisioned pipeline:

User identity → Business permissions → Allowed discoverable tool set → Tool Search → Model → Business execution

This integrates the Agent into the familiar enterprise Java backend system rather than leaving it as a standalone model wrapper.