From 22% Error Rate to 1.3%: How I Made an AI Agent Reliably Handle Order Queries and Refunds

1. Why Many Function Call Demos Lose Control in Production

In the past two years most teams connect a large language model (LLM) directly to business systems and register a few tools such as query_order, refund_order, search_policy, and create_ticket. During the demo phase this looks promising: a user says “show me my order” and the model calls query_order; a user says “I want a refund” and the model calls refund_order. However, once the system is exposed to real traffic the following problems appear.

Model sometimes extracts the wrong parameters because user context and order context are not explicitly bound.

Identical utterances may trigger different tools due to drifting tool‑selection strategies and lack of hierarchical governance.

Duplicate refund requests occur because there is no idempotency key, state machine, or deduplication.

If a tool fails, the model fabricates an answer instead of propagating the failure.

Peak traffic causes latency spikes and error‑rate increases due to excessive downstream dependencies, long serial calls, and missing concurrency isolation.

Post‑mortems are impossible because there is no tool‑call audit, trace propagation, or replay capability.

Many teams blame the model’s intelligence, but the real issue is treating Function Call as a simple “formatted output” rather than a production‑grade execution protocol.

When an AI Agent needs to reliably handle strongly constrained business tasks such as order queries or refunds, the challenge is turning Function Call from a model feature into an auditable, extensible, replayable, and governable production capability.

2. Conclusion: Function Call Solves “How to Call”, Not “How to Execute Stably”

Function Call’s value is that it lets the model express “which tool to call and with what parameters” in a structured way, bridging natural language to machine actions. However, it does not automatically solve production concerns such as tool exposure, authorization, parameter validation, idempotency, concurrency, or observability.

Should the tool be exposed to the model?

Is the call over‑privileged?

Are parameters valid, complete, and bound to the current user?

Do write operations need approval, idempotency, retry, or compensation?

Is concurrent tool usage allowed and are there ordering dependencies?

After a downstream timeout, should we retry, degrade, or hand over to a human?

Can the whole task be audited and replayed?

Thus, from an architectural perspective, Function Call is merely a protocol layer on the execution side, not the entire Agent system.

3. Problem Background: Why Order Query and Refund Are Typical Test Cases

3.1 Order Query Looks Simple but Is Error‑Prone

User may say “show me the order” without providing an order number.

Multiple products or orders may be mentioned in the same session.

Users may have sub‑orders, split orders, or reverse orders.

The model may interpret “latest order” as the latest paid order, while the business wants the latest shipped order.

The order service returns machine fields that must be explained to the user without altering their meaning.

Therefore, order query, although read‑only, involves context binding, structured explanation, and permission checks.

3.2 Refund Is a High‑Risk Write Operation

Refund may require verification of order status, payment status, fulfillment status, and risk status.

Refund can consist of multiple stages: request, approval, channel processing, success, or compensation.

Duplicate execution can cause double refunds, duplicate tickets, or accounting inconsistencies.

“I want a refund” does not always mean immediate execution; sometimes the user is only asking for policy.

Consequently, refund handling requires intent recognition, read/write segregation, state‑machine support, idempotency, compensation, and human approval.

4. Typical Failure Path: Why the Error Rate Soared to 22%

4.1 A Typical Failure Chain

User input: “I want to refund the headphones I bought last week and also check the logistics.”

Model mixes “check logistics” and “apply refund” into a single tool call.

Without an order number, the model guesses the most recent order.

The model triggers a refund without checking sign‑off, return window, or other constraints.

When the refund tool fails, the model still generates a natural‑language confirmation “Your refund has been submitted”.

The frontend displays this fabricated success to the user.

The user sees a success message, but the backend never actually performed the refund – an execution‑consistency error.

4.2 Five Core Root Causes

Tool exposure too wide : All tools are directly visible to the model, causing accidental misuse.

Schema only validates format : JSON schema compliance does not guarantee business‑level correctness.

Lack of explicit state machine : Refund is treated as a single RPC, losing intermediate states.

Retry treated as reliability : Blind retries on write operations cause duplicate actions.

Missing observability and evaluation loop : Failures are only noticed after user complaints; there is no tool‑call audit, sample set, or replay capability.

5. Correct Direction: Incorporate Function Call into a Four‑Plane Architecture

The system should be split into four logical planes:

Control Plane : Decides how a request should be routed.

Execution Plane : Hosts the LLM inference and tool‑call runtime.

State Plane : Persists session, task, and business state.

Governance Plane : Handles tracing, metrics, replay, alerts, and evaluation.

The key insight is that stability is determined not by whether the model calls a tool, but by whether the tool call is wrapped by control, state, and governance layers.

5.1 Control Plane: Decide First, Then Expose Tools

Based on the user utterance, the control plane classifies intent and selects a minimal tool subset for the model. Example mapping:

User expression                Control decision          Tools exposed to model
---------------------------------------------------------------
"Refund will arrive when?"   Rule‑consultation         Only knowledge‑base query
"Can I refund this order?"   Eligibility check        query_order, query_policy, refund_estimate
"I want to refund ORD1001"    High‑risk write          apply_refund (with approval/idempotency)

This dynamic exposure prevents the model from seeing dangerous write tools in low‑risk scenarios.

5.2 Execution Plane: Model as a Controlled Orchestrator

Compose the prompt.

Invoke the LLM.

Parse tool_calls.

Dispatch each call to the Tool Gateway.

Feed observations back to the model.

When stop conditions are met, generate the final answer.

This loop behaves like a runtime rather than a simple chat interface.

5.3 State Plane: Make a Session a Recoverable Task

Session state – user‑level context (order IDs, product hints).

Task state – which steps have been executed.

Business state – refund aggregate, approval status, channel status.

5.4 Governance Plane: Make Errors Visible, Evaluatable, Replayable

Audit every tool call (who, when, which tool, arguments, result).

Distinguish whether an error originates from the model or the tool.

Identify which scenarios are suitable for Function Call versus knowledge‑base lookup.

Detect hot tools that become bottlenecks.

6. Overall Architecture Diagram (Textual)

ClientWeb / App / IM → Agent API → Intent Router → Planner → Function Call Runtime → Tool Gateway →
   Order Service / Refund Service / Policy Service / Logistics Service / Approval Service
   ↳ Session Store (Redis) ↳ Task Store (MySQL) ↳ Refund Aggregate (MySQL) ↳ Outbox ↳ MQ / Event Bus ↳ Refund Worker ↳ Payment Channel
   ↳ Trace / Metrics / Audit

7. Flow Knowledge: Complete Order‑Query Request Path

Key points:

Bind user ID first, then fetch recent orders; let the model pick the target order from a candidate list, reducing cross‑user leakage.

Never let the model output an answer before receiving the tool observation.

Even read‑only operations must be audited (who queried which order, which tool was used, success/failure).

8. Flow Knowledge: Refund Request Split into Eligibility Check and Execution

Refund is divided into two phases:

Eligibility judgment.

Refund application execution.

This prevents “consult‑only” utterances from triggering actual refunds and vice‑versa.

9. Core Design One: Tools Must Be Tiered

Tool Level      Type               Example                     Governance Policy
--------------------------------------------------------------------------------
READ_ONLY      Read‑only tool     query_order, query_logistics  Direct exposure, but with user‑binding and audit
WRITE_GUARDED  Controlled write   apply_refund                  Must have idempotency key, business validation, persisted state
HUMAN_APPROVAL High‑risk tool    large‑amount refund           Requires explicit approval before execution

The control plane dynamically trims the visible tool set based on the current intent.

10. Core Design Two: Schema Is Only the Starting Point

10.1 Structural Validation

JSON schema guarantees field presence and type, e.g.:

{
  "type": "object",
  "properties": {
    "orderNo": {"type": "string"},
    "reason": {"type": "string"},
    "requestId": {"type": "string"}
  },
  "required": ["orderNo", "reason", "requestId"]
}

10.2 Parameter Normalization

Free‑form reasons are mapped to standard enums:

"不想要了" → NO_LONGER_NEEDED "拍错了" → WRONG_PURCHASE "与描述不符" → NOT_AS_DESCRIBED "到货太慢" →

DELIVERY_DELAY

10.3 Business Validation

Order ownership verification.

Order status check.

After‑sale window validation.

Check for existing in‑process refunds.

Risk‑control limits.

Automatic‑refund amount thresholds.

Only after passing all three layers is the write path allowed.

11. Core Design Three: Refund Must Be State‑Machine‑Based

INIT → ELIGIBLE_CHECKING → REJECTED
      → PENDING_APPROVAL → APPROVED → SUBMITTING_CHANNEL → CHANNEL_PROCESSING → SUCCESS
      → FAILED → COMPENSATING → CLOSED

Modeling the refund as a state machine enables idempotent replay, compensation, manual takeover, and reliable recovery for intermediate states such as “local success, channel unknown”.

12. Production‑Grade Code Example: Controlled Tool Gateway

12.1 Spring AI Tool Declaration

import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Component;

@Component
public class OrderTools {
    @Tool(name = "get_order_detail",
          description = "Query order details by order ID. Only use when orderId is explicitly provided or when the context already contains candidate orders.")
    public OrderDetailResponse getOrderDetail(@ToolParam(description = "Order ID, e.g., ORD1001") String orderId) {
        return orderService.queryOrder(orderId);
    }

    @Tool(name = "get_logistics_info",
          description = "Query logistics info for an order.")
    public LogisticsResponse getLogisticsInfo(@ToolParam(description = "Order ID") String orderId) {
        return logisticsService.track(orderId);
    }

    @Tool(name = "apply_refund",
          description = "Submit a refund request. Use only after explicit user intent, eligibility check, and business validation.")
    public RefundResponse applyRefund(@ToolParam(description = "Order ID") String orderId,
                                     @ToolParam(description = "Standardized reason code") String reasonCode,
                                     @ToolParam(description = "Optional comment") String comment) {
        return refundService.apply(orderId, reasonCode, comment);
    }
}

12.2 Dynamic Function Routing (Control Plane)

@Service
public class FunctionRouter {
    private static final Map<String, List<String>> INTENT_TO_TOOLS = Map.of(
        "order_query", List.of("get_order_detail", "get_logistics_info"),
        "refund_apply", List.of("get_order_detail", "apply_refund"),
        "refund_policy", List.of("search_refund_policy"),
        "general", List.of("search_refund_policy")
    );

    public List<String> route(String userQuery) {
        String intent = classifyIntent(userQuery);
        return INTENT_TO_TOOLS.getOrDefault(intent, INTENT_TO_TOOLS.get("general"));
    }

    private String classifyIntent(String query) {
        if (query.contains("申请退款") || query.contains("我要退款") || query.contains("帮我退")) {
            return "refund_apply";
        }
        if (query.contains("退款不到账") || query.contains("能不能退") || query.contains("退款规则")) {
            return "refund_policy";
        }
        if (query.contains("订单") || query.contains("物流") || query.contains("快递")) {
            return "order_query";
        }
        return "general";
    }
}

12.3 Runtime Loop (Execution Plane)

@Service
@RequiredArgsConstructor
public class AgentExecutor {
    private final OpenAiApi openAiApi;
    private final ToolRegistry toolRegistry;
    private final FunctionRouter functionRouter;
    private final ToolGateway toolGateway;
    private final ConversationManager conversationManager;

    public String execute(String sessionId, String userMessage) {
        List<Message> history = conversationManager.getHistory(sessionId);
        history.add(new UserMessage(userMessage));
        List<String> toolNames = functionRouter.route(userMessage);
        List<ToolDefinition> tools = toolRegistry.getTools(toolNames);
        ChatCompletionRequest request = ChatCompletionRequest.builder()
                .model("gpt-4o")
                .messages(history)
                .tools(tools.stream()
                        .map(t -> Tool.fromFunction(t.name(), t.description(), t.schema()))
                        .toList())
                .toolChoice(ToolChoice.AUTO)
                .build();
        ChatCompletionResponse response = openAiApi.chatCompletion(request);
        return handleResponse(sessionId, response);
    }

    private String handleResponse(String sessionId, ChatCompletionResponse response) {
        Message assistantMessage = response.getChoices().get(0).getMessage();
        conversationManager.addMessage(sessionId, assistantMessage);
        if (assistantMessage.getToolCalls() == null || assistantMessage.getToolCalls().isEmpty()) {
            return assistantMessage.getContent();
        }
        List<ToolResultMessage> results = new ArrayList<>();
        for (ToolCall toolCall : assistantMessage.getToolCalls()) {
            ToolInvocation invocation = ToolInvocation.from(toolCall);
            ToolResult result = toolGateway.invoke(buildToolContext(sessionId), invocation);
            results.add(new ToolResultMessage(toolCall.getId(), result));
        }
        return continueWithToolResults(sessionId, results);
    }
}

12.4 Resilient Tool Execution (Governance Plane)

@Component
public class ResilientToolExecutor {
    private static final int DEFAULT_TIMEOUT_MS = 5000;
    @Autowired
    private CircuitBreakerFactory circuitBreakerFactory;

    public Object execute(String toolName, Map<String, Object> params) {
        CircuitBreaker circuitBreaker = circuitBreakerFactory.create(toolName);
        return circuitBreaker.run(() -> {
            CompletableFuture<Object> future = CompletableFuture.supplyAsync(() -> doExecute(toolName, params));
            try {
                return future.get(DEFAULT_TIMEOUT_MS, TimeUnit.MILLISECONDS);
            } catch (TimeoutException ex) {
                future.cancel(true);
                throw new ToolTimeoutException("tool timeout: " + toolName, ex);
            }
        }, throwable -> getFallbackResult(toolName, params));
    }

    @Retryable(value = {RemoteException.class}, maxAttempts = 3,
               backoff = @Backoff(delay = 1000, multiplier = 2))
    protected Object doExecute(String toolName, Map<String, Object> params) {
        return invokeDownstream(toolName, params);
    }
}

12.5 Tool Definition & Registry

public enum ToolRiskLevel { READ_ONLY, WRITE_GUARDED, HUMAN_APPROVAL }

public record ToolDefinition(String name, String description, ToolRiskLevel riskLevel, Duration timeout, JsonNode schema) {}

public interface ToolExecutor {
    ToolDefinition definition();
    ToolResult execute(ToolInvocation invocation);
}

12.6 Tool Gateway Implementation

@Slf4j
@Component
@RequiredArgsConstructor
public class ToolGateway {
    private final ToolRegistry toolRegistry;
    private final ToolSchemaValidator schemaValidator;
    private final ToolPolicyService toolPolicyService;
    private final ToolAuditService toolAuditService;
    private final ExecutorService toolExecutorPool;

    public ToolResult invoke(ToolContext context, ToolInvocation invocation) {
        ToolExecutor executor = toolRegistry.get(invocation.toolName());
        ToolDefinition definition = executor.definition();
        long start = System.currentTimeMillis();
        String traceId = context.traceId();

        schemaValidator.validate(definition.inputSchema(), invocation.arguments());
        JsonNode normalizedArgs = toolPolicyService.normalizeArguments(definition, context, invocation.arguments());
        toolPolicyService.authorize(context, definition, normalizedArgs);

        Future<ToolResult> future = toolExecutorPool.submit(() -> executor.execute(invocation.withArguments(normalizedArgs)));
        try {
            ToolResult result = future.get(definition.timeout().toMillis(), TimeUnit.MILLISECONDS);
            toolAuditService.recordSuccess(traceId, definition.name(), normalizedArgs, result, System.currentTimeMillis() - start);
            return result;
        } catch (TimeoutException ex) {
            future.cancel(true);
            toolAuditService.recordFailure(traceId, definition.name(), normalizedArgs, "TIMEOUT", System.currentTimeMillis() - start);
            throw new ToolExecutionException("tool timeout: " + definition.name(), ex);
        } catch (Exception ex) {
            toolAuditService.recordFailure(traceId, definition.name(), normalizedArgs, "FAILED", System.currentTimeMillis() - start);
            throw new ToolExecutionException("tool failed: " + definition.name(), ex);
        }
    }
}

12.7 Order Query Tool (Read‑Only)

@Component
@RequiredArgsConstructor
public class QueryOrderTool implements ToolExecutor {
    private final OrderQueryService orderQueryService;

    @Override
    public ToolDefinition definition() {
        return new ToolDefinition("query_order", "Read‑only order query", ToolRiskLevel.READ_ONLY,
                Duration.ofSeconds(2), ToolSchemas.queryOrderSchema());
    }

    @Override
    public ToolResult execute(ToolInvocation invocation) {
        String userId = invocation.context().userId();
        String orderNo = invocation.arguments().path("orderNo").asText(null);
        boolean latest = invocation.arguments().path("latest").asBoolean(false);
        OrderView order = latest ? orderQueryService.queryLatestOrder(userId)
                                 : orderQueryService.queryUserOrder(userId, orderNo);
        if (order == null) {
            return ToolResult.retryableFailure("ORDER_NOT_FOUND", "未找到匹配订单，请用户补充订单号或商品信息");
        }
        return ToolResult.success(Map.of(
                "orderNo", order.orderNo(),
                "status", order.status(),
                "productName", order.productName(),
                "logisticsStatus", order.logisticsStatus()
        ));
    }
}

12.8 Refund Application Tool (Write‑Guarded)

@Component
@RequiredArgsConstructor
public class CreateRefundRequestTool implements ToolExecutor {
    private final RefundApplicationService refundApplicationService;

    @Override
    public ToolDefinition definition() {
        return new ToolDefinition("create_refund_request", ToolRiskLevel.WRITE_GUARDED,
                Duration.ofSeconds(3), ToolSchemas.createRefundSchema());
    }

    @Override
    public ToolResult execute(ToolInvocation invocation) {
        RefundCommand command = RefundCommand.builder()
                .requestId(invocation.arguments().path("requestId").asText())
                .userId(invocation.context().userId())
                .orderNo(invocation.arguments().path("orderNo").asText())
                .reasonCode(invocation.arguments().path("reasonCode").asText())
                .comment(invocation.arguments().path("comment").asText(""))
                .build();
        RefundAcceptance acceptance = refundApplicationService.accept(command);
        return ToolResult.success(Map.of(
                "refundNo", acceptance.refundNo(),
                "status", acceptance.status(),
                "message", acceptance.message()
        ));
    }
}

12.9 Refund Aggregate (State‑Machine Core)

@Getter
public class RefundAggregate {
    private String refundNo;
    private String requestId;
    private String userId;
    private String orderNo;
    private RefundStatus status;
    private int retryCount;

    public static RefundAggregate create(RefundCommand command, String refundNo) {
        RefundAggregate agg = new RefundAggregate();
        agg.refundNo = refundNo;
        agg.requestId = command.getRequestId();
        agg.userId = command.getUserId();
        agg.orderNo = command.getOrderNo();
        agg.status = RefundStatus.PENDING_APPROVAL;
        return agg;
    }

    public void approve() {
        if (status != RefundStatus.PENDING_APPROVAL) {
            throw new IllegalStateException("invalid refund state for approval: " + status);
        }
        status = RefundStatus.APPROVED;
    }

    public void markChannelSubmitting() {
        if (status != RefundStatus.APPROVED && status != RefundStatus.RETRY_PENDING) {
            throw new IllegalStateException("invalid state for channel submit: " + status);
        }
        status = RefundStatus.SUBMITTING_CHANNEL;
    }

    public void markSuccess() {
        if (status != RefundStatus.SUBMITTING_CHANNEL && status != RefundStatus.CHANNEL_PROCESSING) {
            throw new IllegalStateException("invalid state for success: " + status);
        }
        status = RefundStatus.SUCCESS;
    }

    public void markRetryPending() {
        retryCount++;
        status = RefundStatus.RETRY_PENDING;
    }
}

12.10 Application Service with Idempotency

@Service
@RequiredArgsConstructor
public class RefundApplicationService {
    private final RefundRepository refundRepository;
    private final RefundEligibilityService refundEligibilityService;
    private final RefundOutboxService refundOutboxService;

    @Transactional
    public RefundAcceptance accept(RefundCommand command) {
        RefundAggregate existing = refundRepository.findByRequestId(command.getRequestId());
        if (existing != null) {
            return RefundAcceptance.of(existing.getRefundNo(), existing.getStatus().name(), "幂等命中，返回已有退款申请");
        }
        refundEligibilityService.verify(command.getUserId(), command.getOrderNo(), command.getReasonCode());
        String refundNo = RefundNoGenerator.nextNo();
        RefundAggregate aggregate = RefundAggregate.create(command, refundNo);
        refundRepository.save(aggregate);
        refundOutboxService.appendRefundAcceptedEvent(aggregate);
        return RefundAcceptance.of(refundNo, aggregate.getStatus().name(), "退款申请已受理");
    }
}

12.11 Outbox Table (SQL)

CREATE TABLE refund_outbox_event (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    event_id VARCHAR(64) NOT NULL,
    aggregate_type VARCHAR(32) NOT NULL,
    aggregate_id VARCHAR(64) NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    payload_json JSON NOT NULL,
    status VARCHAR(16) NOT NULL,
    retry_count INT NOT NULL DEFAULT 0,
    next_retry_at DATETIME NULL,
    created_at DATETIME NOT NULL,
    updated_at DATETIME NOT NULL,
    UNIQUE KEY uk_event_id (event_id)
);

13. High‑Concurrency & Scalability: Where the System Gets Stressed

LLM quota limits.

Thread pools.

Downstream RPC connection pools.

Redis/QPS limits.

Database hotspot rows.

Message‑queue backlog.

13.1 Four Common Bottlenecks

Model call bottleneck : Multi‑round Function Calls increase latency and cost. Mitigate by reducing unnecessary clarifications, caching rule answers, using lightweight intent classifiers, and externalizing fixed workflows.

Tool gateway bottleneck : Serial execution makes any slow downstream drag the whole chain. Parallelize independent reads (order, logistics, policy) when there is no write conflict.

State storage bottleneck : Hot orders or users can lock rows. Split aggregate tables from audit tables, shard Outbox scanning, throttle hot users.

Asynchronous execution bottleneck : Refund channel spikes can cause duplicate consumption or dead‑letter buildup. Use idempotent keys, lease‑based workers, and dead‑letter handling.

13.2 Multi‑Level Concurrency Control Model

Level          Control Object                Strategy
-----------------------------------------------------------------
User‑level     Single‑user instantaneous request   Throttle bots, prevent spam
Tenant‑level   Business‑line quota                Prevent a single tenant from exhausting shared Agent resources
Tool‑level     query_order / create_refund_request  Independent rate‑limit and circuit‑breaker per tool
Downstream‑level Payment channel, order service   Bulkhead isolation of connection pools
Task‑level     Asynchronous refund task           Lease + exponential backoff + max retries

13.3 Scaling Considerations

Session state must be shared (Redis) rather than in‑process. Task and business state must be persisted in a shared DB. Write tasks need distributed lease or DB optimistic lock to avoid duplicate execution across multiple workers.

14. Reliability Design: Failure Is Part of the Main Flow

14.1 Timeout Controls (per tool)

Order query: 300‑800 ms.

Policy lookup: 500‑1500 ms.

Refund acceptance: 1‑3 s.

Channel refund: asynchronous, not in front‑end timeout budget.

Write operations that depend on slow downstream should not promise synchronous success.

14.2 Retry Strategy Matrix

Scenario                     Retry?   Reason
----------------------------------------------------------
Read service occasional timeout   Yes   Low‑impact, idempotent
Write service network timeout      Cautious  Check idempotent result first
Payment channel definite failure   No   Record failure, trigger compensation
Parameter / permission error      No   Immediate user feedback

14.3 Degradation Principles

When order service is flaky, respond “system busy, please retry later”.

If policy service is unavailable, fall back to human hand‑off or static policy page.

Channel refund spikes: accept request, process asynchronously.

Model failure: fallback to rule router or human support.

14.4 Human Takeover & Replay

For high‑value refunds or disputes, the system must allow operators to replay a request using the stored traceId, view the tool set exposed at that moment, inspect generated parameters, and see the final answer.

15. Observability & Evaluation: Closing the Loop

15.1 Essential Metrics

Session request volume, success rate, error rate.

Tool call success, timeout, retry rates.

P50/P95/P99 latency per tool.

Refund creation success, channel refund success.

Clarification, hand‑off, degradation rates.

15.2 Trace Fields

{
  "traceId": "trc_20260713_0001",
  "sessionId": "cs_9001",
  "userId": "U1001",
  "toolName": "create_refund_request",
  "riskLevel": "WRITE_GUARDED",
  "arguments": {"orderNo":"ORD1001","reasonCode":"NOT_AS_DESCRIBED","requestId":"req_8d9a"},
  "resultCode": "ACCEPTED",
  "latencyMs": 148,
  "timestamp": "2026-07-13T10:30:21Z"
}

15.3 OpenTelemetry Instrumentation

@Configuration
public class ObservabilityConfig {
    @Bean
    public OpenTelemetry openTelemetry() {
        return OpenTelemetrySdk.builder()
                .setTracerProvider(sdkTracerProvider())
                .setMeterProvider(sdkMeterProvider())
                .build();
    }
    @Bean
    public Tracer tracer(OpenTelemetry ot) { return ot.getTracer("agent-runtime"); }
}

@Component
@RequiredArgsConstructor
public class InstrumentedToolExecutor {
    private final Tracer tracer;
    private final MeterRegistry meterRegistry;

    public Object execute(String toolName, Map<String, Object> params) {
        Span span = tracer.spanBuilder("tool.execute." + toolName)
                .setAttribute("tool.name", toolName)
                .setAttribute("tool.params", params.toString())
                .startSpan();
        try (Scope ignored = span.makeCurrent()) {
            long start = System.currentTimeMillis();
            Object result = doExecute(toolName, params);
            long duration = System.currentTimeMillis() - start;
            meterRegistry.timer("tool.execution.duration", "tool", toolName).record(duration, TimeUnit.MILLISECONDS);
            meterRegistry.counter("tool.execution.count", "tool", toolName, "status", "success").increment();
            span.setStatus(StatusCode.OK);
            return result;
        } catch (Exception ex) {
            meterRegistry.counter("tool.execution.count", "tool", toolName, "status", "error").increment();
            span.recordException(ex);
            span.setStatus(StatusCode.ERROR);
            throw ex;
        } finally {
            span.end();
        }
    }
}

15.4 Evaluation Sample Construction

Correct tool selection.

Correct parameter extraction.

Proper clarification when uncertain.

Avoid accidental write triggers.

Graceful degradation on tool failure.

Final answer matches actual business state.

Maintain three test sets: single‑intent samples, mixed‑intent samples, and adversarial samples.

16. Deployment & Elastic Scaling

16.1 Deployment Example (Kubernetes)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-agent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-agent
  template:
    metadata:
      labels:
        app: ai-agent
    spec:
      containers:
      - name: agent
        image: registry.example.com/ai-agent:latest
        ports:
        - containerPort: 8080
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: openai-secret
              key: api-key
        - name: REDIS_HOST
          value: redis-service
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

16.2 Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-agent-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-agent
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

16.3 Scaling Tips

Separate synchronous API pods from asynchronous worker pods.

Isolate tool‑gateway thread pools from LLM call pools.

Health checks protect traffic shifting.

Scale on custom metrics such as tool‑call timeout rate, queue depth, and thread‑pool saturation, not just CPU.

Gray‑release high‑risk tool versions before full rollout.

17. Path from 22% to 1.3%: What to Do First, What Later

17.1 Phase 1 – Stop Bleeding

Tier read/write tools; hide refund execution tools from casual Q&A.

Enforce idempotency keys and audit logs on all writes.

Prevent the model from fabricating success when a tool fails.

17.2 Phase 2 – State‑Machine Refund

Replace the single RPC with a multi‑step flow: eligibility → acceptance → state persistence → Outbox → async channel → callback → final state.

17.3 Phase 3 – Observability & Evaluation Loop

Implement full tool‑call audit, trace propagation, and replay.

Build error sample sets and automated evaluation.

17.4 Phase 4 – High‑Concurrency Optimizations

Cache rule answers, parallelize independent reads.

Introduce worker lease, exponential backoff, and tenant‑level throttling.

Bulkhead downstream services.

18. Common Pitfalls: If Not Fixed, Error Rate Will Rise Again

Assuming schema validation alone guarantees correctness.

Feeding the model ever more context; instead, expose only the minimal tool set.

Treating blind retries as reliability for writes.

Letting the model decide on high‑risk tools without rule‑engine, amount thresholds, risk checks, or approval workflow.

Using the final natural‑language reply as the source of truth instead of the persisted business state.

19. Applicability Boundary

19.1 When This Approach Is Suitable

Scenarios involving orders, payments, refunds, tickets, or any strong business constraints.

Mixed read/write conversational flows.

Need for auditability, replayability, and governance.

Multi‑tenant, permission‑sensitive, or approval‑required use cases.

High request volume where async processing and scaling matter.

19.2 When Not to Over‑Engineer Immediately

Pure internal demos.

Read‑only FAQ bots with no write side‑effects.

Very low traffic prototypes.

Business rules still in flux; premature heavy architecture adds friction.

In early stages keep tool tiering, audit, and idempotency, then evolve to full state‑machine and async pipelines.

20. Production Checklist

20.1 Architecture Checks

Control, execution, state, and governance planes are separated.

Read/write tools are tiered.

High‑risk writes are decoupled from the synchronous path.

20.2 Engineering Checks

All write tools have idempotency keys.

Tool Gateway enforces validation, authorization, timeout, and audit.

No path allows a failed tool to produce a fabricated success.

Outbox, de‑duplication, and compensation mechanisms are in place.

20.3 High‑Concurrency Checks

User‑level, tenant‑level, and tool‑level rate limiting.

Isolation and circuit‑breakers for hotspot downstream services.

Workers support lease, backoff, and dead‑letter handling.

Session and task state are stored in shared stores (Redis, DB) to allow stateless scaling.

20.4 Governance Checks

Full replayability via traceId.

Evaluation suites covering single intent, mixed intent, and adversarial cases.

Monitoring of tool success, timeout, and degradation rates.

Manual takeover paths for large refunds, disputes, or risk triggers.

21. Final Takeaway

Function Call solves the problem of “how the model reaches a tool”, but a production‑ready system must answer “how the system reliably executes the tool”. The five decisive factors are:

Control‑plane intent routing before exposing tools.

Separate governance for read‑only and write‑heavy operations.

Model refunds as a state machine rather than a single RPC.

Use Outbox, idempotency, and async workers to protect intermediate states.

Close the loop with audit, replay, evaluation, and alerting.

Reducing error rates is not about longer prompts or bigger models; it is about embedding the Agent inside a mature software‑engineering framework.

Function Call solves how the model reaches the system; a production architecture solves how the system sustains the model.

If your Agent is still at the “model + a few registered functions” stage, the next most valuable investment is to add the control, state, and governance planes before piling on more features.