How to Perform a Graceful Shutdown of an Elasticsearch Node

In Elasticsearch daily operations, gracefully shutting down a node—e.g., for disk upgrade, memory expansion, version upgrade, or decommission—requires a series of steps to avoid service interruption and data loss.

1. Verify cluster configuration and avoid split‑brain. Before removing a node, check the number of master‑eligible nodes and the discovery.zen.minimum_master_nodes setting. For small clusters the recommended value is (master_eligible_nodes / 2) + 1. Example configuration:

discovery.zen.minimum_master_nodes: 3
node.master: true

If the setting needs to be changed, it can be updated dynamically:

# Set minimum_master_nodes to 2
curl -XPUT 'http://hostname:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent" : {
    "discovery.zen.minimum_master_nodes" : 2
  }
}'

2. Exclude the node from the cluster routing allocation. Use the transient setting cluster.routing.allocation.exclude._name (or _ip, _host) to tell Elasticsearch to move shards away from the target node:

curl -XPUT http://hostname:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d '{
  "transient": {
    "cluster.routing.allocation.exclude._name": "{node.name}"
  }
}'

The exclusion triggers the shard allocation mechanism; shards will relocate to other nodes, which may take several minutes but does not affect client requests.

3. Wait for shard relocation to finish. Monitor the process with commands such as:

curl -s "http://hostname:9200/_cat/shards" | grep RELOCATING
curl http://hostname:9200/_cluster/health?pretty

When all shards have moved, verify that the node holds no documents or store size:

curl http://hostname:9200/_nodes/{node.name}/stats/indices?pretty
# look for "docs": {"count": 0, "deleted": 0} and "store": {"size_in_bytes": 0}

4. Stop the node process. After shard migration, stop the Elasticsearch service or kill the process:

# If managed by systemd
systemctl stop elasticsearch.service
# Or kill the PID directly
kill {pid}

5. Restore the cluster routing strategy. Remove the exclusion rule so the node can be added back later:

curl -XPUT http://hostname:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d '{
  "transient": {
    "cluster.routing.allocation.exclude._name": null
  }
}'

At this point the node has been gracefully taken out of the cluster. To bring it back, simply start the service; Elasticsearch will automatically re‑join and receive shard allocations.

Summary : By checking minimum_master_nodes, excluding the node from allocation, waiting for shard relocation, stopping the service, and finally clearing the exclusion, administrators can safely perform a smooth Elasticsearch node shutdown without disrupting writes or queries.