AI Customer Service at Millions Scale: Chat API to Function Calling (SpringBoot)

Why a simple chat API fails in production

In an e‑commerce support scenario with more than 12 million daily sessions, user queries often require real‑time data from order, after‑sale, coupon or operations systems. A minimal /chat loop that only forwards the user message to a large language model (LLM) can generate fluent text, but it cannot access authoritative data, cannot safely execute actions, and will hallucinate when the answer is unknown. This turns a business risk into a reliability problem.

Evolution stages

Stage 1 – Pure /chat

Receive user message

Compose system prompt and conversation history

Call the LLM

Return the answer

Works for demos and pure FAQ, but has three hard limits: no real‑time data, no safe execution of actions, and hallucination when the model does not know the answer.

Stage 2 – Prompt‑driven JSON

Teams try to force the model to emit a fixed JSON structure, e.g.:

{
  "intent": "QUERY_ORDER",
  "orderId": "ORD20231201"
}

The JSON is then parsed and used to call a backend service. In production this quickly breaks because:

The model does not always emit syntactically valid JSON.

Multiple intents cannot be expressed with a single static schema.

Prompt length is consumed by format constraints, leaving little room for business semantics.

When many capabilities are needed, maintaining the prompt becomes unmanageable.

Stage 3 – Spring AI Function Calling

Spring AI introduces a clean contract: the model decides *which* function to call and supplies parameters; the application performs permission checks, parameter validation, execution, and returns a structured result that the model uses to generate the final natural‑language reply. This split raises reliability because the unstable part (model reasoning) stays within the model, while the stable part (business execution) stays in code.

Stage 4 – New operational complexity

Model may select the wrong function.

Model may pass malformed parameters.

Function latency can block the chat thread.

Write operations can be invoked multiple times.

Long function results can exhaust the token budget.

Downstream failures can be “masked” by the model if not handled explicitly.

How Spring AI Function Calling works (order‑status example)

User sends “Did my order ORD20231201 ship?”

The chat service sends the user message, conversation history, and the list of currently exposed functions to the LLM.

The LLM returns a function‑call intent, e.g. queryOrder(orderId=ORD20231201).

Spring AI matches the intent to the queryOrder bean and executes it.

The bean validates the user, checks ownership, calls the order service, and returns a structured response (status, message, etc.).

The response is fed back to the LLM, which composes a fact‑based natural‑language answer.

The crucial point is that the model only performs intent understanding and language generation; all business credibility is guaranteed by the application code.

Key implementation pieces

Data models – Java record s define request and response schemas, making the contract explicit for the model.

public record OrderQueryRequest(String orderId) {}
public record OrderQueryResponse(String orderId, String status, String message) {}
public record RefundRequest(String orderId, String reason) {}
public record RefundResponse(String status, String refundId, String message) {}

Function beans – each bean validates parameters, checks permissions, handles exceptions, and returns a structured error code when needed.

@Bean
@Description("Query order status for the current user.")
public Function<OrderQueryRequest, OrderQueryResponse> queryOrder(OrderServiceFeignClient orderClient) {
    return request -> {
        String userId = SecurityContextHolder.getContext().getAuthentication().getName();
        if (request == null || request.orderId() == null || request.orderId().isBlank()) {
            return new OrderQueryResponse(null, "INVALID_ARGUMENT", "Missing order ID");
        }
        try {
            if (!orderClient.verifyOwner(request.orderId(), userId)) {
                return new OrderQueryResponse(request.orderId(), "FORBIDDEN", "No permission");
            }
            return orderClient.queryOrder(request.orderId());
        } catch (Exception ex) {
            return new OrderQueryResponse(request.orderId(), "SYSTEM_ERROR", "Order service busy, try later");
        }
    };
}

@Bean
@Description("Apply refund for the current user.")
public Function<RefundRequest, RefundResponse> applyRefund(RefundService refundService) {
    return request -> {
        String userId = SecurityContextHolder.getContext().getAuthentication().getName();
        if (request == null || request.orderId() == null || request.orderId().isBlank()) {
            return new RefundResponse("INVALID_ARGUMENT", null, "Missing order ID");
        }
        return refundService.submit(userId, request.orderId(), request.reason());
    };
}

Chat service routing – decides which functions are visible for the current turn, preventing the model from seeing unrelated capabilities.

@Service
public class IntelligentChatService {
    private final ChatClient chatClient;
    private final ConversationMemoryService memoryService;

    public IntelligentChatService(ChatClient.Builder builder, ConversationMemoryService memoryService) {
        this.chatClient = builder.build();
        this.memoryService = memoryService;
    }

    public String chat(String sessionId, String userId, String userMessage) {
        List<String> candidateFunctions = routeFunctions(userMessage);
        List<Message> history = memoryService.load(sessionId);
        String answer = chatClient.prompt()
            .system("""
                You are an e‑commerce AI assistant.
                For order, after‑sale, or coupon queries, you must call the provided functions.
                Never fabricate results.
                If a function fails, tell the user honestly.
                """)
            .messages(history)
            .user(userMessage)
            .functions(candidateFunctions.toArray(String[]::new))
            .call()
            .content();
        memoryService.appendRound(sessionId, userMessage, answer);
        return answer;
    }

    private List<String> routeFunctions(String message) {
        if (message.contains("退款") || message.contains("售后")) {
            return List.of("queryOrder", "applyRefund");
        }
        if (message.contains("订单")) {
            return List.of("queryOrder");
        }
        return List.of();
    }
}

Tool audit context – records every function call (trace id, session id, user id, tool name, arguments, result code, snapshot, timestamp) for post‑mortem analysis.

public final class ToolAuditContext {
    private static final ThreadLocal<List<ToolCallRecord>> TOOL_CALLS = ThreadLocal.withInitial(ArrayList::new);
    public static void record(String toolName, Map<String, Object> arguments, String result) {
        TOOL_CALLS.get().add(new ToolCallRecord(toolName, arguments, result, Instant.now()));
    }
    public static List<ToolCallRecord> snapshot() { return List.copyOf(TOOL_CALLS.get()); }
    public static void clear() { TOOL_CALLS.remove(); }
}

Operational concerns

Timeouts, retries and circuit breaking – read‑only functions use short timeouts, minimal retries and cache fall‑backs; write functions enforce stricter retries, idempotency keys, and may fall back to asynchronous confirmation.

Idempotency and audit – write operations record an idempotency_key (e.g., userId+orderId+type) and a business‑unique constraint in the database. Redis locks reduce contention but do not replace database constraints.

Conversation memory and token budgeting – instead of a fixed number of rounds, the system caps total token usage. Recent rounds are kept verbatim, older rounds are summarized, and long function results are stored as concise summaries to avoid context_length_exceeded errors.

Scaling considerations – at tens of millions of daily sessions the bottleneck shifts from LLM latency to downstream services (order, refund, coupon). Horizontal scaling of the chat service, Redis hot‑key management, and asynchronous audit logging (RocketMQ/Kafka) become essential.

Best‑practice patterns

Dynamic function exposure – expose only the minimal capability set required for the current intent (e.g., knowledge lookup only, order query only, refund only).

Second‑layer validation – even if the model suggests a call, the function must re‑validate ownership, parameter legality, order status, duplicate submission, etc.

Never expose all functions globally – a large function pool increases mis‑selection risk and makes the prompt larger.

Audit logs – capture trace_id, session_id, user_id, tool_name, arguments, result_code, result_snapshot, and timestamps for every call.

Token budget enforcement – prune history by total token count, keep recent rounds full, summarize older rounds, and truncate long function results.

Failure handling – if a function fails, the model must report the failure directly instead of fabricating a successful answer.

When to adopt Function Calling

Pure FAQ or knowledge‑base scenarios – a simple /chat endpoint is sufficient.

Read‑only data look‑ups (order status, coupon validity) – introduce read‑only Function Calling.

Write‑heavy actions (refunds, order changes) – mandatory controlled Function Calling with permission checks, idempotency, and audit.

Common pitfalls

Assuming a successful function call guarantees a correct final answer – the model can still mis‑phrase the response.

Registering all functions globally – increases the chance of mis‑selection.

Relying solely on Redis locks for write safety – must enforce database constraints and idempotency records.

Keeping only the last 10 rounds – function results can consume tokens faster than user messages, leading to context_length_exceeded.

Model may try to call a non‑existent function – enforce a whitelist of allowed functions per turn.

Production checklist

Dynamic per‑intent function exposure.

Parameter and permission validation inside each function.

Separate timeout/retry policies for read vs. write functions.

Idempotency keys and business‑unique constraints for high‑risk writes.

Comprehensive function‑call audit logs with traceability.

Token budget enforcement for conversation history and function results.

Failure‑scenario testing (service timeouts, missing functions, duplicate submissions, cache loss).

Ensure failed function calls are reported as failures, not masked as successes.