Huolala Tech

Jun 20, 2024 · Backend Development

Why Our Core Service Crashed: Tomcat Thread Pool Bottlenecks & Custom Executor Pitfalls

On October 27, 2023, a sudden surge in request volume and downstream latency caused the bfe‑customer‑application‑query‑svc to exhaust its Tomcat thread pool, triggering health‑check failures and pod restarts; a deep dive revealed that an ill‑designed custom executor and unchecked thread‑waiting calls created a bottleneck that amplified the outage.