A Two‑Day AI‑Built SaaS Triggers a Bug That Burns a Month’s Server Costs in One Day

Jumpei Ueno, a senior cloud‑infrastructure engineer, took over a SaaS project that the company’s CFO had built in two days using Claude Code. While reviewing the LLM API cost chart, he saw a single day’s spend spike to nearly half of the month’s total, exceeding the entire server‑cluster cost for the month.

His first hypothesis was that the CFO’s rapid development caused many manual test runs, each invoking expensive LLM APIs. However, log analysis of the task queue, database, and request records revealed that the same high‑cost batch was being executed repeatedly by the program itself—21 times for the same tenant—rather than by human operators.

The batch workflow calls several LLMs, then writes the results to a database. The failure occurred in the second step: the production database lacked a newly added column because the migration had not yet run, so the write raised a “column does not exist” error and the task ended with a 500 response.

All LLM requests returned 200 and were fully billed; only the final database write failed, causing the task to be marked as failed and retried. Ueno likened this to finishing a full restaurant meal, paying the bill, then falling and ordering the same meal again, repeating the charge.

This situation exemplifies a “Retry Storm”: instead of retrying failed requests, the system repeatedly re‑executed successful LLM calls because the task was not idempotent and the queue automatically retried on any 500 error.

Two root causes overlapped: (1) deployment order error—code was deployed before the database migration, creating a deterministic failure that never self‑corrects, and (2) the task‑queue’s automatic retry mechanism, which kept re‑invoking the entire batch despite the underlying schema issue.

From the incident, Ueno distilled several lessons:

Deterministic failures should not be retried endlessly. Errors such as missing schema or 4xx responses will not be fixed by repeated attempts; the system should abort and surface the problem.

High‑impact tasks must be idempotent. Operations that incur real costs, like LLM API calls, should skip already‑completed work on retry.

Deploy database migrations before code that depends on new fields. This eliminates the window where code accesses nonexistent columns.

Monitor cost‑related APIs and separate production/test API keys. Without visibility, excessive charges may go unnoticed until the bill arrives.

The post concludes that while AI tools dramatically lower the barrier for non‑engineers to create functional products, understanding the hidden failure modes and cost implications remains a specialized engineering responsibility.