In high‑concurrency systems, tail latency is driven primarily by waiting on locks, resources, and scheduling rather than raw computation, with phenomena like head‑of‑line blocking, context‑switch overhead, and cache‑coherency costs amplifying unpredictable delays.
This article explains the hedged request pattern used by Google to combat microservice tail latency, shows how to implement it in Go with context and goroutines, discusses its impact on latency and load, and explores additional techniques such as SingleFlight and service‑class prioritization to further reduce tail delays.
The article details how Didi’s large‑scale Ceph deployment suffered from high tail latency due to long‑held and coarse‑grained locks, and describes a series of fixes—including asynchronous read threads, fine‑grained object caches, per‑thread lock‑free logging, and lock‑free filestore apply—that cut latency by up to 90 % and more than doubled read throughput.
This article explains why a tiny fraction of slow requests can dominate overall service latency, explores their causes—especially slow nodes on the query path—and shows how proper tracing and analysis can improve response times and cut infrastructure costs.
