Coder Trainee

Oct 20, 2026 · Artificial Intelligence

How 2,000 Orders Turned an 800ms AI Agent into an 8-Second Disaster

An AI customer service agent's P99 latency spiked from 800ms to 8.2s when a user with 2,147 orders triggered an unbounded tool that returned 1.8MB JSON (400k tokens), causing the model to hallucinate; fixes included limiting results to 5 orders with 4 fields, adding timeouts, and implementing fallback responses, restoring P99 to 890ms.