Why One Agent Works But 1,000 Fail: Capacity Planning for Agent Orchestration

Task Not Finished Doesn't Mean Execution Resources Needed Continuously

A procurement assistant may read attachments, call a model, query a supplier system, then wait hours for user confirmation. Keeping a browser and worker process idle during that wait exhausts resources. The solution: save progress, results, and pending confirmations, then release the environment; re-acquire resources when the user replies. Some browser sessions are hard to restore after closing, so retain them with an idle timeout and a maximum count instead of holding them indefinitely. On recovery, verify incomplete actions — browser recycling only means the environment disappeared, not that prior business requests failed.

All Called a Task, But Resource Consumption Varies Greatly

"How many agents does the platform support?" is not a precise capacity question. Reading text and driving a browser cost differently; waiting for approval and batch model calls cannot share a single concurrency limit. The article breaks down workload types:

Document analysis & model calls — consumes model quota, context, request concurrency. Capacity design must answer: how to queue after rate limiting, do long contexts crowd out short tasks?

Browser or code sandbox — consumes memory, CPU, sessions, temporary storage. Questions: startup latency, idle time before reclamation?

Business tool queries — consumes downstream connections, API quotas. Question: if downstream slows, should the platform keep expanding calls?

Waiting for approval — consumes persistent state, notifications, expiry management. Question: does it still unnecessarily occupy an executor?

Parallel subtasks — parent task spawns many children, multiplying total resource use. Question: what is the maximum work a single task can derive?

Record arrival rate, execution time, wait time, and resource usage per task class before deciding concurrency. Measuring only model latency does not reveal total platform capacity.

Unacceptable Work Must Be Handled at Entry

If downstream is already rate-limited, launching more tasks only adds timeouts and retries. Use bounded queues: assign resources to runnable work, queue the rest, but cap queue length. When full or wait too long, explicitly tell the caller to delay, cancel, or reject — this is backpressure. Queues cannot be infinite; queued procurement materials may expire, so re-validate timeliness on dequeue. For multi-tenant platforms, limit each tenant's concurrency and max wait time, reserve resources for critical work, and enforce priority rules so not every task becomes "highest priority."

Quotas, Permissions, and Isolation Are Three Separate Concerns

Limiting a team to ten sandboxes addresses resource usage; restricting data access to their own addresses authorization; preventing sandbox escape addresses execution isolation. A Kubernetes namespace only groups resources — network connectivity, credential sharing, and host resource limits must be checked separately. Kubernetes multi-tenancy documentation discusses how quotas combine with these isolation measures; a single config item is insufficient. Parent tasks must account for child tasks: each child may stay within quota, but unbounded spawning can bypass limits. Check both task-total budget and tenant-total budget, not just per-process.

After Cancellation, Resources and Actions Must Have a Destination

On user cancel: stop scheduling queued work, let running work respond to cancellation at allowed points, clean up browsers and temp files. But cancellation is not time travel — a submitted purchase request cannot be marked "never happened" just because the local task stopped. Use a restricted reconciliation process to determine actual state, then let business rules decide next steps. For lost worker processes, require periodic lease renewal; on expiry reclaim execution rights. However, the old process may still be alive but network-partitioned. The new process taking over must not blindly redo: business interfaces need deduplication, execution identity, or version checks to prevent duplicate submissions.

Reconciliation, cleanup, and recovery also need reserved capacity; otherwise the busier the platform, the harder these housekeeping tasks become.

Load Testing Must Include Adverse Conditions

Prioritize verifying: massive cold starts, sustained model throttling, downstream latency spikes, sudden tenant bursts, and worker exits after external actions commit. Observe not just completion count but queue latency, cross-tenant interference, cancellation effectiveness, resource reclamation, and duplicate actions. Intentionally push to rejection to confirm graceful self-protection. At small scale, a complex scheduler may be unnecessary — limit queue length and concurrency, persist task state well, and iterate based on observed bottlenecks. First ensure the platform under overload can report queue status and wait bounds, avoids cascading downstream failures, and reclaims resources on cancel. Only then does discussing maximum concurrency have basis. The next article will cover per-task budget constraints.

References

Kubernetes Multi-tenancy: https://kubernetes.io/docs/concepts/security/multi-tenancy/