In high‑concurrency systems, tail latency is driven primarily by waiting on locks, resources, and scheduling rather than raw computation, with phenomena like head‑of‑line blocking, context‑switch overhead, and cache‑coherency costs amplifying unpredictable delays.
Netflix discovered that scaling hundreds of containers on modern CPUs hit severe lock‑contention due to mount‑related kernel locks, with performance varying across AWS instance types, NUMA designs, and hyper‑threading, leading them to redesign containerd mounting and choose hardware‑aware scheduling to restore efficient scaling.
The article explains how the traditional mmap_lock read lock creates severe contention and priority inversion for frequent madvise MADV_DONTNEED/FREE calls, and how a per‑VMA locking redesign eliminates this bottleneck, improves performance, and is slated for the next Linux kernel release.
The article explains how replacing the global mmap_lock read lock with per‑VMA locks for madvise MADV_DONTNEED/FREE removes severe lock contention and priority inversion in Linux memory management, and it details the patches, reasoning, and performance evidence supporting the change.
This article examines the severe row‑lock competition in MySQL during hot‑SKU inventory positioning, analyzes its causes and risks, and presents a comprehensive set of mitigation strategies—including DB account isolation, flow‑control, request merging, and message‑queue optimizations—validated by extensive performance testing and successful rollout across multiple clusters.
This article investigates a production issue on Java 21 where virtual threads cause Tomcat to stop processing requests, closeWait sockets to accumulate, and locks to deadlock, detailing the diagnostics, thread‑dump analysis, lock inspection, and the eventual conclusions about virtual‑thread behavior.
An in‑depth postmortem of a costly MySQL data‑migration failure caused by using INSERT INTO SELECT without proper indexing, detailing the OOM issue, full‑table scans, lock contention, and best practices to safely employ the statement.
The article explains that while MySQL foreign keys ensure data consistency and integrity, they introduce performance overhead, lock contention, and scalability issues—especially in high‑concurrency and sharded environments—leading many large internet firms to forego them in favor of application‑level solutions.
The article examines the hidden lock‑contention issue in InnoDB’s Adaptive Hash Index (AHI) observed during high‑concurrency sysbench runs, explains why multiple threads block on the AHI hash table’s exclusive lock, and presents a lightweight fallback optimization that checks the lock before building AHI, reducing lock wait time and stabilizing QPS performance.
The article details how Didi’s large‑scale Ceph deployment suffered from high tail latency due to long‑held and coarse‑grained locks, and describes a series of fixes—including asynchronous read threads, fine‑grained object caches, per‑thread lock‑free logging, and lock‑free filestore apply—that cut latency by up to 90 % and more than doubled read throughput.
This article outlines a systematic approach to diagnosing and resolving DDL execution hangs in Dble, covering log inspection, context analysis, identifying problematic data nodes and MySQL connections, and confirming lock contention as the root cause.
The article investigates intermittent slow update performance in a Tencent Cloud internal system caused by massive lock_wait_thread contention, analyzes the underlying InnoDB lock mechanisms and thread behavior, implements a fix by disabling lock_wait_suspend_thread triggers, and demonstrates substantial latency reduction through benchmark results.
An in‑depth investigation reveals that when MySQL semi‑sync replication is configured with rpl_semi_sync_master_wait_for_slave_count=1, starting a second slave can cause the master’s dump thread to fail, leading to slave_io_thread stalls; the analysis includes reproduction steps, status checks, thread stack traces, and a patch using sched_yield to resolve the lock contention.
This article shares years of server‑development experience, explaining how data copies, context switches, memory allocation, and lock contention act as the four main performance killers and offering practical strategies to mitigate each for faster, more scalable services.
