How to Unlock Elasticsearch Write Block and Fix Logstash 403 Errors

Phenomenon: Kafka backlog causes Logstash to fail writing logs to Elasticsearch.

Logstash error:

[logstash.outputs.elasticsearch][main][] Retrying failed action {:status=>403, :error=>{"type"=>"cluster_block_exception", "reason"=>"index [] blocked by: [FORBIDDEN/8/index write (api)];"}}

Cause: Kibana dev tools show the index setting index.blocks.write: true, which locks the index and prevents writes.

Solution: Remove the write block so the index can accept writes.

Method 1 (CLI): Execute on the Elasticsearch server:

curl -uusername:password -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.write": null}'

Method 2 (Kibana Dev Tools):

PUT /_all/_settings
{
  "index.blocks.write": null
}