CFO’s Two‑Day Claude Code Launch Leads to a Bug That Makes One Day of AI Calls Cost a Month of Server Fees

Cost anomaly detection

A single day’s LLM API usage accounted for nearly half of the month’s total bill, exceeding the monthly server‑cluster cost.

Investigation

Log analysis of the task queue, database, and request records showed that a batch processing task for one tenant was executed 21 times on the same day.

Batch task workflow

Issue a series of requests to multiple LLMs (the primary cost source).

Write the returned results into the database.

Root cause

Deterministic failure: The code expected a new column that had not yet been added by the production database migration, causing a “column does not exist” error and a 500 response.

Automatic retry: The managed task queue treats any 500 response as a transient fault and automatically retries the task.

Non‑idempotent design: The batch job does not check whether work has already been completed, so each retry re‑issues all LLM calls.

Retry storm effect

All LLM calls returned HTTP 200 and were billed. Because the failure occurred after the successful calls, the retry mechanism re‑executed the entire batch. Each retry generated a full new round of LLM charges, resulting in 21 executions and a massive cost spike.

Key takeaways

Deterministic failures (e.g., schema mismatches or 4xx errors) should not be retried indefinitely; retries must have a clear limit and abort on known program errors.

Tasks with side effects that incur charges must be idempotent or skip already‑completed work.

Production deployments should follow the order “migrate database first, then deploy code” to avoid windows where code expects a schema that does not yet exist.

Cost visibility (e.g., monitoring API‑cost charts) is essential to detect abnormal spending early.

Code example

往
期
推
荐
1、
十万个why：Nginx 已经能做负载均衡，为什么还需要服务注册发现？
2、
Claude Agent 进入 IntelliJ IDEA 公测，Java 开发方式真的要变了
3、
新来的CTO：禁止在项目中使用 Date 类，发现立即走人！！！
4、
从 JDK 21 升到 25 代码全红？复盘 Java 史上最难产的 API
5、
为什么这么多设备都禁ping？禁ping到底图什么？
6、
不是干掉 Node：Nub 想做的是 Node.js Plus
点
分
享
点
收
藏
点
点
赞
点在看