How to Diagnose and Resolve CPU/Memory Spikes in Agent Batch Processing

1. Problem Analysis

An Agent processes documents one by one without issues, but when importing 100,000 documents at once, CPU spikes and memory hits the limit, causing the container to be OOM killed. Reducing model call concurrency from 100 to 10 does not stop memory growth.

The root cause likely occurs before model invocation: all documents are loaded into memory, 100,000 async tasks are created, each holding the original text, parsed results, and pending prompts. The model rate limiter only guards a small execution slice, not the entire data lifecycle.

The key is not tuning a single thread count but making the system explicit about: how many tasks may expand simultaneously, how much data each stage may retain, and where upstream must pause when processing lags.

1.1 Control the Expanding Working Set First

Batch execution risk comes from too many simultaneously live objects, not necessarily memory leaks. A single file may exist as compressed bytes, then decompressed text, then parsed object tree, then chunks, then vectors, then tool results — multiple representations coexist, making the per-task working set far larger than the original file size.

Agents further expand tasks: 100 active tasks each spawning 8 parallel tool calls could theoretically create 800 sub-calls. An entry limit of 100 does not bound total in-flight operations. Each sub-call's response body, logs, and retry state add resident memory.

Emergency mitigation: tighten new batch admission, broker prefetch, and task dispatch to let running work drain; isolate online and offline workloads into separate resource pools; keep accepted tasks in reliable queues or task tables — never drop them to reduce memory. Cancellation must propagate down the call chain, confirming exit and permit release before resuming dispatch.

If the service can still be sampled, preserve performance evidence from the peak. A simple restart often wipes the scene; resuming consumption will re-expand the same batch and the fault returns.

1.2 Distinguish Compute Hotspots from Resource Waits

High CPU in offline computing is not inherently a failure; the question is whether throughput keeps growing and latency/reliability targets hold. CPU percentage must specify the denominator: per-core, whole machine, or container quota — avoid mixing metrics with different bases.

High CPU requires answering where time is spent. CPU profiles can pinpoint document parsing, JSON codec, tokenizer, local embedding, or sorting hotspots; if GC share rises, examine allocation rate and live heap to judge whether massive temporary objects are being continuously created, copied, and collected.

Containers add another case: the process needs more CPU but has exhausted its cgroup quota. Check cgroup throttling metrics — CPU flame graphs cannot show time the process never got. Kubernetes CPU limits throttle via CFS; memory limits trigger OOM kill — they are different protections. The Kubernetes resource management documentation explains this distinction.

The hotspot bars and memory curves in the diagram illustrate diagnostic directions, not measured data.

Memory diagnosis requires aligning the time windows of task slowdown, queue growth, and resource peaks. Go's inuse_space tracks live heap objects; alloc_space is cumulative allocation — use time-differencing to gauge allocation pressure. Neither equals process RSS or container memory directly. See Go pprof documentation for sampling details.

A drop after batch completion usually indicates a transient working-set spike; a rising baseline across repeated loads warrants checking caches, reference chains, goroutine leaks, and native allocations — not declaring a leak from one curve. Heap dropping while RSS stays high may reflect runtime memory retention.

These factors amplify each other: CPU throttling slows processing, objects are held longer; more in-flight data increases GC pressure, further stealing business compute time. Therefore, investigation must not treat CPU and memory as unrelated dashboards.

1.3 Determine Concurrency by Resource Budget

Worker count cannot be set by task count alone. Capacity planning must break out baseline, execution working set, and buffers:

Peak memory budget ≈ baseline model & cache
  + Σ(active tasks per stage × per-task peak working set)
  + Σ(queue & unpersisted result resident bytes)
  + runtime overhead & safety margin

Each resident datum counts once; objects that finish a CPU stage but are still held downstream cannot be subtracted early. Estimates need representative load tests plus hard per-item size limits — average file size is insufficient.

Example: a 4 GiB container, minus 1 GiB baseline & margin, 0.5 GiB queue & results budget, leaves 2.5 GiB for active tasks. If homogeneous tasks peak at 128 MiB working set, memory-side candidate concurrency is floor(2560 / 128) = 20. These numbers illustrate the calculation, not a recommended or validated configuration.

That 20 is further constrained by CPU and downstream capacity. For the same task type and stage, convert each constraint to a concurrency budget and take the minimum — do not directly min CPU cores, GiB, and RPM. CPU side: estimate sustained CPU seconds per task, then load-test to find parallelism. Available cores × target utilization gives CPU-seconds budget per second, not instantaneous concurrency.

When task sizes vary widely, use weighted permits or size-based pools. Large documents consume more memory budget, small texts less; cap per-tenant and per-batch shares. Tasks whose weight exceeds total pool capacity must be pre-split or diverted, else they may never acquire a permit. Estimated weights are not enough — enforce byte limits during actual read and decompress.

1.4 Make Backpressure Span Input to Output

A common mistake: create all goroutines first, then acquire a semaphore inside. Even if only 10 run, the rest already hold stacks, closures, and referenced objects. The correct control point is before creating heavy tasks or loading large objects, or simply use fixed workers pulling from a bounded queue.

Python async has a similar pitfall: passing all coroutines to asyncio.gather and using a Semaphore inside only limits the execution window, not the number of created Tasks or the final result collection. TaskGroup provides structured lifecycle management but does not auto-limit task count. The Python asyncio documentation describes these scheduling and result-collection behaviors.

Therefore, input must use cursors, pagination, or lazy iteration; queues should carry task IDs, object storage references, and lightweight metadata. Switching to a Reader is not enough if the parser then builds a full object tree — the peak remains.

The whole chain can be split into: bounded pending queue, CPU processing pool, tool call pool, bounded result queue, persistence stage. When a downstream queue fills, upstream must pause production — not switch to another unbounded temporary list. Each layer limits both count and bytes, and total residency across all layers is accounted.

Broker prefetch is part of this boundary. RabbitMQ's prefetch limits unacked message count per consumer; multiple consumers' residency adds up, and it is not a message byte limit. With 16 workers but prefetching thousands of large messages, memory can still saturate. RabbitMQ prefetch documentation gives exact semantics.

Task expansion budget must cover root tasks, tool fan-out, and retries, constraining both instance and cluster totals. A parent waiting for children should not hold all permits the children need; release budget in stages or let a coordinator dispatch. Otherwise a full pool of parents waiting for unstartable children turns concurrency limiting into deadlock.

1.5 Isolate Compute-Intensive Stages

Remote model and tool calls spend most time waiting on network; local parsing, vector compute, and compression continuously occupy CPU. Mixing them in one high-concurrency pool lets compute stages burst and starve the processor. Therefore, set separate CPU parallelism and in-flight I/O limits, pass results between stages, and release no-longer-needed data.

Check for hidden secondary parallelism: outer 8 workers, inner inference library spawning multiple threads per session — the OS sees layered thread pools. ONNX Runtime's intra-op threads, inter-op parallelism, and spinning all affect CPU consumption; configure with session reuse and real load tests, not just language-level worker counts. ONNX Runtime thread management documentation provides these knobs.

Go's GOMAXPROCS controls simultaneous Go code execution parallelism, not total goroutines or all native threads. From Go 1.25 the default can sense Linux cgroup CPU quota, but actual behavior still depends on Go version, module compatibility settings, and explicit config — cannot assume it always equals container CPU limit. Go runtime documentation states these boundaries.

After hotspot confirmation, apply targeted optimizations: reduce duplicate parsing of the same document, eliminate unnecessary JSON round-trips, reuse read-only model instances. Merging embedding requests amortizes overhead, but micro-batches must still respect count, token, and byte budgets; larger batches are not always faster.

1.6 Shorten Data Residency Windows

Streaming input is only half the battle. If each completed item's full result is appended to an in-memory array for a final report, memory still grows with total task count. Safer: persist each item, keep only result IDs and status in a bounded window, move historical indexes to a task table, retain minimal in-memory stats; generate final reports on demand or in segments. Result references are small but accumulate to non-constant memory.

When ordered results are required, slow tasks block subsequent ones. The first item stalls, thousands of completed items pile up in a reorder buffer. Ordered output needs a bounded reorder window, or external persistence by sequence number then read-back — never unbounded caching.

If a single task exceeds capacity, dropping concurrency to 1 does not help. Raw file bytes, decompressed bytes, page/pixel count, chunk count, tool output, and execution time all need ceilings; oversize tasks must be sharded or routed to a dedicated large-task pool before heavy processing. Sharding must preserve necessary context and aggregation logic to avoid sacrificing answer completeness for memory savings.

Temporary files or object storage can offload large results, but require quotas, cleanup cycles, and failure recovery. Memory-backed emptyDir uses tmpfs and does not relieve container memory pressure by writing to it. Kubernetes documentation explicitly warns this.

Finally, audit logs, trace export queues, HTTP response bodies, and exception objects to prevent them from re-referencing entire documents. Large buffers should not grow unbounded in reuse pools, or caches built to reduce allocation will instead raise the long-term baseline.

1.7 Runtime Parameters Are Only a Safety Net

Go's GOMEMLIMIT is a runtime soft limit, not equal to process RSS or container hard limit, and does not cover all native memory. It cannot reclaim data still referenced by tasks; setting it too low causes frequent GC. GOGC similarly trades memory for GC CPU, not a knob that lowers both. First reduce working set and allocations, then tune parameters. The Go GC guide explains these trade-offs.

Scale out only after per-instance working set is controlled. New replicas load models, fill caches, start prefetching, and collectively consume downstream quotas; more replicas do not increase model service quota. During recovery, raise admission and prefetch gradually, preserve global concurrency and retry budgets, use backoff with jitter for retries, keep side-effect operations idempotent.

1.8 Verify Peak Memory Stops Growing with Total Volume

The most convincing test: fix the in-flight budget, increase task count from 10k to 100k. Processing time may grow, but memory peak should be capped by the residency window, not scale linearly with total items. This directly exposes full pre-read, premature task creation, and centralized result aggregation.

Then incrementally raise in-flight budget, observing effective throughput, queue time, end-to-end P95, CPU throttling, GC, live heap, and container memory. If concurrency rises but throughput plateaus while latency degrades, you are at or past the capacity knee — keep a safety margin instead of pushing CPU utilization higher.

Acceptance must also mix in oversized documents, high decompression ratios, multi-tool fan-out, slow writes, observability backend failures, and cancellation/retry storms. Verify that queue bytes, reorder windows, and thread counts stay bounded; after recovery, tasks must not be lost, duplicate side effects, or hold permits indefinitely. The final standard: quality and completion rate do not regress, resources stay in an acceptable steady state — not just lower monitoring curves.

2. Reference Answer

I would first throttle new batches, prefetch, and task dispatch; isolate online and offline resource pools; preserve accepted tasks; avoid repeated restarts that re-expand the same batch. Then align the peak time window: use CPU profiles to inspect parsing, local inference, and GC; use heap and allocation data to examine working sets; simultaneously check container CPU throttling and native threads. Transient memory spikes are not necessarily leaks — they may simply be too much simultaneously live data.

The core redesign is a bounded pipeline from input through execution to output: cursor-based paginated reads, acquire permits before creating heavy tasks, fixed workers processing, queues with dual count-and-byte limits, backpressure propagated upstream when queues fill. Concurrency is derived from memory working set, CPU, and downstream capacity, covering tool fan-out and retries; CPU-intensive and I/O stages use separate pools, including internal inference threads. Results are persisted per item, large objects are size-limited or sharded early, ordered output uses bounded reorder windows. GOMEMLIMIT and scaling are only auxiliaries. Finally, with fixed in-flight budget, scale total task count to verify memory no longer grows linearly with volume, then stress-test with bursts, slow downstreams, and cancellation/retry to determine safe capacity — ensuring no task loss, no quality degradation.