Disk Watermark

vivo Internet Technology

Nov 22, 2023 · Operations

Investigation and Resolution of Elasticsearch node_concurrent_recoveries Performance Issue

The team traced read‑request timeouts to a single overloaded Elasticsearch node where an excessively high node_concurrent_recoveries setting caused many simultaneous shard recoveries and disk‑watermark‑driven relocations, and resolved the issue by lowering concurrent recoveries, enabling adaptive replica selection, and adjusting allocation settings.