I Integrated AI into Our Java System in One Week — The Token Bill Almost Killed Me

Day 1: I Took the Wrong Path

Given the requirement to add AI Q&A to an existing customer service system, the author's first instinct was to use Spring AI Alibaba because its Graph orchestration looked powerful enough for an "enterprise AI platform." The first day stalled — not because the framework was bad, but because the author overcomplicated the solution.

The actual customer service needs were only three things:

User asks "Where is my order?" → call order query API

User asks "How to return?" → query knowledge base

User abuses → appease + transfer to human

Yet the initial design drew a Graph workflow containing "intent recognition node → knowledge retrieval node → tool scheduling node → multi-turn dialogue management node." The author deleted it all.

On day two, the approach switched to the simplest thing: one ChatClient + three Tools. Half a day later it worked.

@Bean
ChatClient customerServiceClient(ChatClient.Builder builder, CustomerTools tools) {
    return builder
        .defaultSystem("""
            You are an e-commerce customer service assistant.
            1. Order issues call queryOrder
            2. Refund issues call getRefundPolicy
            3. If unsolvable call transferToHuman
            Keep answers under 100 characters.
            """)
        .defaultTools(tools)
        .build();
}

The lesson: the detour wasn't a framework mistake; it was trying to build a "platform" before solving a "problem."

Day 3: The Bill Scared Me for the First Time

After the function worked, load testing began: simulate 1,000 users, 3 turns each = 3,000 requests. The cloud bill showed nearly ¥200. At that rate, a system with 10,000 daily conversations would burn ¥20,000/month on model calls alone. Most dialogues (greetings, "are you there?") don't need an expensive model.

Three optimizations cut the cost:

1. Route simple queries to a cheaper model.

public ChatModel route(String question) {
    // Greetings, simple queries → cheap model
    if (question.length() < 20 && !needsReasoning(question)) {
        return gpt35;
    }
    // Complex questions → strong model
    return gpt4;
}

2. Compress context. Previously 10 turns of history went into the prompt; now only the last 3 turns + a summary of earlier dialogue.

3. Add semantic caching. Common questions ("how to return", "shipping cost") hit a semantic cache with ~30% hit rate.

Same load test: cost dropped from ¥200 to just over ¥40 — an 80% reduction.

Day 5: Production Incident

Friday afternoon, first day live. At 3 PM monitoring alerted: response time spiked from 800 ms to 8 seconds. The model service was healthy. The culprit was a Tool the author wrote: "query all user orders." One user had 2,000+ historical orders; the Tool fetched all, serialized to JSON, and stuffed it into the model context. The model choked on the huge context for 7 seconds.

Two fixes:

Add result limit to the Tool — default return only the latest 5 orders.

Add timeout circuit breaker — if Tool execution exceeds 2 seconds, return "query timeout, please retry."

After redeploy, response time returned to under 900 ms.

Lesson: writing a Tool for an Agent isn't "works functionally"; treat it like an external API — rate limiting, pagination, timeouts, none can be missing.

Day 7: Three Extras at Delivery

The week ended, but the author spent two more days adding three things not in the original spec:

1. Token budget alert.

@Scheduled(fixedDelay = 60000)
public void checkBudget() {
    long todayTokens = getTodayTokenUsage();
    if (todayTokens > DAILY_BUDGET * 0.8) {
        alertService.send("Token consumption reached 80% of daily budget", todayTokens);
    }
}

2. Conversation logging. Every user question and AI answer persisted — not for audit, but to later tune prompts and run evaluations with real data.

3. A kill switch. One toggle to instantly fall back to human agents when things break, without code changes or redeploy.

The boss didn't ask for these, but the author knew: without them, the feature wouldn't survive a month.

Closing Thoughts

The biggest takeaway after a week wasn't "AI is amazing," but: embedding AI into a running Java system isn't technically hard; the hard part is making it "keep running sustainably." Cost, timeouts, rate limiting, fallbacks, observability — these have nothing to do with AI itself, but decide whether AI survives in production. And that's exactly what Java developers excel at.

# Practical config cheatsheet
# Model routing: simple → gpt-3.5, complex → gpt-4
# Context compression: last 3 turns + summary
# Semantic cache: ~30% hit rate
# Tool design: rate limit + pagination + timeout
# Must-have trio: budget alert + conversation logging + kill switch