CFO’s No‑Code AI App Triggers a Month’s Server Bill in One Day

Background

Jumpei Ueno, a senior cloud‑infrastructure engineer, inherited a project that was not written by a professional developer but by the company’s CFO. Using Claude Code, the CFO created and launched a SaaS product in just two days and then handed over maintenance to Ueno.

Cost Spike Discovery

While reviewing the LLM API cost chart, Ueno noticed that one day's AI‑call expense was unusually high—almost half of the entire month’s API spend. He realized that the AI‑call cost for that single day exceeded the monthly operating cost of the whole server cluster.

Initial Investigation

Ueno first asked the CFO what had been done on that day, but the CFO could not recall any specific action. Ueno then began a systematic investigation.

Wrong Hypothesis: Repeated Manual Testing

His initial hypothesis was that the CFO’s rapid development and repeated testing in production caused many expensive LLM calls. The commit history showed over twenty commits related to the AI feature on that day, which seemed to support this idea.

Actual Cause: System‑Level Retry Storm

Log analysis of the application, task queue, and database revealed a different picture. The same high‑cost batch task was being executed repeatedly by the system—21 times for the same tenant. Humans would not press a button 21 times in a day; the program was.

Task Workflow

Send a series of requests to multiple LLMs (the primary cost source).

Write the model responses to the database.

The failure occurred in step 2: the code referenced a new column that had not yet been added by the database migration, so the database threw a “column does not exist” error and the task ended with a 500 response.

All LLM requests returned 200 status codes and were billed; only the final database write failed, causing the task to be marked as failed.

Analogy and Definition of “Retry Storm”

Ueno likened the situation to finishing a full restaurant meal, paying the bill, then stumbling, forgetting the meal, and ordering the whole set again—repeating the costly process 21 times. Although “Retry Storm” usually describes repeated failed requests, here the calls succeeded but were retried because the system discarded the successful result and started over.

Root Causes

1. Deployment Order Error : Code was deployed before the database migration, so the new column did not exist in production. This deterministic failure would not be resolved by retries.

2. Automatic Retry Mechanism : The managed task queue automatically retried any task that returned a 500 error, assuming a transient fault. Because the failure was due to a missing column, retries kept re‑executing the entire non‑idempotent batch, causing 21 full LLM call cycles and duplicate billing.

Lessons Learned

Deterministic failures (e.g., schema mismatches or 4xx errors) should not be retried indefinitely; retries must have a clear limit and should abort on such errors.

High‑impact tasks that incur real costs must be designed to be idempotent, allowing the system to skip already‑completed work.

Production deployments must follow the order “migrate database first, then deploy code” to avoid a window where code expects a schema that does not yet exist.

If cost signals are not observable, the problem may only be discovered after the money is spent. Monitoring API keys, budget alerts, and separate test‑environment keys can surface anomalies early.

Conclusion

Building a feature in two days is feasible, but preventing “well‑intentioned automatic retries” from turning successful LLM calls into hidden, repeated charges requires operational expertise that cannot be learned in a weekend.