How 2,000 Orders Turned an 800ms AI Agent into an 8-Second Disaster

Last Wednesday at 2:40 PM, a monitoring alert showed the AI customer service agent's P99 response time had jumped to 8.2 seconds. The previous day's load test had shown 800ms — a 10x degradation.

Root Cause: An Overlooked Tool Returning Unbounded Data

The agent architecture used a ChatClient with three tools: queryOrder, queryRefundPolicy, and transferToHuman. Slow-request logs revealed that every request exceeding 8 seconds was an order inquiry where the user did not provide an order number.

Typical user queries:

"Where's the stuff I bought last week?"

"Why haven't my recent orders shipped?"

"Show me the orders I placed."

The queryOrder tool required an orderId. When users omitted it, the model would call another tool — getUserOrders — to fetch all orders for that user and then pick the relevant one.

@Tool(description = "根据订单号查询订单状态")
public String queryOrder(String orderId) {
  Order order = orderRepository.findByOrderNo(orderId);
  return JSON.toJSONString(order);
}

A Three-Year Customer with 2,147 Orders

The getUserOrders implementation had no pagination, no limit, and no field trimming:

public String getUserOrders(Long userId) {
   List<Order> orders = orderRepository.findByUserId(userId);
   return JSON.toJSONString(orders);
}

For the problematic user, 2,147 orders with a dozen fields each serialized to ~1.8MB JSON — nearly 400k tokens. Processing that context plus inference took 8 seconds. Worse, the model, overwhelmed by data, couldn't locate the target order and called queryOrder again with a guessed ID, making a second tool call with the same massive context.

Three Fixes That Restored Sub-Second Latency

1. Hard Limit on List Tools

Renamed to getUserRecentOrders, now returns only the 5 most recent orders with four fields (orderNo, productName, status, createTime):

@Tool(description = "查询用户最近的订单列表，最多返回 5 条。仅当用户未提供订单号、且需要从最近订单中定位具体订单时调用")
public String getUserRecentOrders(Long userId) {
   List<Order> orders = orderRepository.findTop5ByUserIdOrderByCreateTimeDesc(userId);
   List<OrderBrief> briefs = orders.stream()
     .map(o -> new OrderBrief(o.getOrderNo(), o.getProductName(), o.getStatus(), o.getCreateTime()))
     .toList();
   return JSON.toJSONString(briefs);
}

Token count dropped from 400k to under 200.

2. Timeout Wrapper for Every Tool Call

Added a 2-second timeout; on timeout, return a human-readable message so the model can pivot instead of hanging:

public String executeWithTimeout(ToolCallback tool, String args, long timeoutMs) {
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<String> future = executor.submit(() -> tool.call(args));
try {
return future.get(timeoutMs, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
future.cancel(true);
return "查询超时，请稍后重试或提供订单号直接查询";
} finally {
executor.shutdownNow();
}
}

The point of timeouts isn't to make things faster — it's to prevent a single slow tool from blocking the entire request.

3. Explicit Fallback Responses for Tool Failures

Each tool now returns a conversational message on error, not null or an exception:

@Tool(description = "根据订单号查询订单状态")
public String queryOrder(String orderId) {
try {
Order order = orderRepository.findByOrderNo(orderId);
if (order == null) {
return "未找到该订单号的订单，请向用户确认订单号是否正确";
}
return JSON.toJSONString(order);
} catch (Exception e) {
log.error("订单查询失败, orderId={}", orderId, e);
return "订单查询服务暂时不可用，请告知用户稍后重试";
}
}

Fallback isn't "return empty" — it's "return a sentence the model understands and can continue the conversation with."

Post-Fix Metrics (Same Load Test: 1,000 Users, 3 Turns Each, Including the 2,147-Order User)

P99 Response Time: 8.2 s → 890 ms

Avg Tokens per Request: 120k → 3,800

Daily Tool Timeouts: 200+ → 0

Model Hallucination Rate: 15% → 3%

The hallucination drop is notable: when the tool returns too much data, the model loses focus and starts fabricating. With only 5 recent orders and 4 fields, accuracy improved — the model now correctly says "Your recent order is in transit, estimated delivery tomorrow."

Closing Principle

This incident reframed a core principle:

Writing tools for an agent isn't designing APIs — it's writing manuals for the model.

In API design, "return all data" is lazy but often harmless. In agent scenarios, that laziness directly becomes latency, cost, and hallucinations.

Pagination, timeouts, fallbacks — these used to be "backend concerns." Now it's clear: if an interface is called by a model, none of the three can be missing.

# Three takeaways you can copy
# 1. Every list tool must have a hard limit + field projection
# 2. Every tool call must have a timeout; on timeout return human-readable text, not an exception
# 3. Tool error handling must return a message the model can act on, not null