Understanding Elasticsearch Disk Watermarks in 3 Minutes

Basic Concept

Elasticsearch uses a disk watermark mechanism to monitor each node's disk usage. By setting low, high, and flood‑stage thresholds, the cluster can automatically take actions such as stopping new writes or relocating shards when space runs low.

Importance

Prevent data loss : Monitoring disk usage avoids loss caused by insufficient space.

Maintain system stability : When disk space is low, Elasticsearch may stop writing new data, which could degrade performance or interrupt services.

Improve data recovery : Early warnings and preventive actions make recovery easier before data is lost.

How It Works

The watermarks are based on disk usage percentages:

Low Watermark : Default 85%. When reached, Elasticsearch stops allocating new primary shards to the node (existing shards remain).

High Watermark : Default 90%. When reached, Elasticsearch attempts to move existing shards off the node, affecting all shard allocations.

Flood Stage : Default 95%. When reached, the node is marked as non‑allocatable, all shards are moved elsewhere, and the indices on the node become read‑only. This is the last safeguard against disk exhaustion.

Configuration Method

Set the thresholds in elasticsearch.yml:

cluster.routing.allocation.disk.watermark.low: 85%</code>
<code>cluster.routing.allocation.disk.watermark.high: 90%</code>
<code>cluster.routing.allocation.disk.watermark.flood_stage: 95%

Or use absolute values:

cluster.routing.allocation.disk.watermark.low: 200g</code>
<code>cluster.routing.allocation.disk.watermark.high: 100g</code>
<code>cluster.routing.allocation.disk.watermark.flood_stage: 30g

Best Practices

Regular monitoring : Frequently check watermark status and overall cluster health to ensure sufficient free space.

Backup strategy : Implement reliable backups to protect against data loss.

Optimize storage : Use efficient storage solutions such as SSDs to improve performance and reduce space consumption.

Plan shards wisely : When creating indices, choose appropriate numbers of primary shards and replicas to limit disk usage.

Clean unused data : Periodically delete obsolete data to free up space.