Why Unlimited Token Budgets Let Agents Waste Money on Retries

Why cost is easily underestimated

When an agent encounters an error it automatically retries, rewrites prompts, and calls tools, causing costs to accumulate without notice. Long loops stack token usage, model pricing, tool fees, and waiting time.

AI cost should be accounted per successful task and include not only tokens but also retries, tool invocations, waiting time, and manual verification. Many expenses are hidden in failed re‑runs, waiting, storage, and external tools.

Billing should be split into four parts

Compute and token consumption for normally completed tasks.

Additional consumption generated by retries, rollbacks, and duplicate tool calls.

Platform costs for preserving state, logs, and evaluation data.

Human time spent inspecting erroneous results and handling exceptions.

Often only per‑minute API limits are set, without a total budget for a single task.

Budget must be bound to the task

Define budgets for token count, call count, wall‑clock time, and external fees; pause execution once any limit is exceeded. The MCP roadmap and OpenTelemetry semantic specifications are pushing standardisation of agent identity, execution chains, and observability data.

Global rate‑limiting protects the platform but cannot judge whether a specific task is still worth continuing. Task‑level budgets align better with business value and make anomalies easier to explain.

What to do after exceeding the budget

Pausing does not equal failure. The system can save a checkpoint, indicate what has been completed and what is missing, and let a human decide whether to continue, degrade, or terminate.

The model sets the capability ceiling, while the harness sets the production floor. The longer the task and the more powerful the tools, the higher the importance of surrounding engineering.

Critical tasks may request a higher budget, but the reason and expected benefit must be recorded.

Don’t forget long‑term maintenance costs

New architecture introduces objects that persist long term: configurations, catalogs, compatible versions, state tables, monitoring, alerts, and cleanup jobs. An initial deployment of one or two instances can become an entire platform after months.

Set per‑task budgets for token usage, tool calls, duration, and retries. When limits are hit, downgrade or hand over to human handling; agents should not be allowed to self‑correct indefinitely.

Solution comparisons must consider coordination effort, fault investigation time, onboarding learning curve, and version retirement, not just development speed.

Approval interfaces should display impact scope, key parameters, and irreversible consequences; a single “confirm” button makes quality decisions difficult.

If costs lack a clear owner, developers and operations often cover them with overtime. Embedding maintenance responsibility in the design is more practical than vague collaboration slogans.

Architects deliver judgment criteria, not just components

Components will evolve and product names will change. The more durable artifact is the judgment framework: which layer the problem occurs in, which state must be reliably persisted, where degradation is allowed, and which actions must be performed by humans.

Long loops stack token, model price, tool fees, and waiting time. If a team can only repeat implementation steps without explaining this logic, they will repeat the same pitfalls when the technology changes.

After a design review, ask: if the component is replaced in six months, can today’s boundaries, metrics, and recovery processes still be used? The reusable parts become the true architectural assets.

This article is original content by IT Architecture. Unauthorized reproduction is prohibited.