Mastering Elasticsearch Index Lifecycle Management with Docker Compose

Introducing Index Lifecycle Management

In daily Elasticsearch management, large volumes of log and behavior data grow over time; only recent (hot) data is frequently accessed while older (cold) data is rarely used, so without proper lifecycle policies storage and performance suffer.

Common ILM Phases

hot : index receives many read/write operations.

warm : index is read‑only but still queried.

cold : index has few reads and no writes.

delete : index can be safely removed.

Note: phases can be customized per business needs.

Deploying an Elasticsearch Cluster

Deploy a cluster with 2 hot, 2 warm, and 2 cold nodes, plus Kibana and Cerebro for monitoring. The docker-compose.yml is:

version: '2.2'
services:
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: hwc_cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts.0.host=http://elasticsearch:9200
    networks:
      - hwc_es7net
  kibana:
    image: docker.elastic.co/kibana/kibana:7.1.0
    container_name: hwc_kibana7
    environment:
      - XPACK_GRAPH_ENABLED=true
      - TIMELION_ENABLED=true
      - XPACK_MONITORING_COLLECTION_ENABLED="true"
    ports:
      - "5601:5601"
    networks:
      - hwc_es7net
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    container_name: es7_hot1
    environment:
      - cluster.name=cr7-hwc
      - node.name=es7_hot1
      - node.attr.box_type=hot
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2
      - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - hwc_es7data_hot1:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - hwc_es7net
  # (additional warm and cold nodes omitted for brevity)
volumes:
  hwc_es7data_hot1:
    driver: local
  hwc_es7data_warm1:
    driver: local
  hwc_es7data_cold1:
    driver: local
networks:
  hwc_es7net:
    driver: bridge

Run docker-compose up -d to start the cluster (Docker required).

Adjusting ILM Poll Interval

Set the ILM poll interval to 1 second for demonstration:

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval":"1s"
  }
}

Defining ILM Policies

Create a policy with hot, warm, cold, and delete phases:

hot : rollover after 5 documents.

warm : after 20 s make index read‑only and allocate to warm nodes.

cold : after 40 s allocate to cold nodes and set replicas to 0.

delete : after 60 s delete the index.

PUT /_ilm/policy/log_ilm_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": { "rollover": { "max_docs": 5 } }
      },
      "warm": {
        "min_age": "20s",
        "actions": {
          "allocate": { "include": { "box_type": "warm" } },
          "readonly": {}
        }
      },
      "cold": {
        "min_age": "40s",
        "actions": {
          "allocate": { "include": { "box_type": "cold" } },
          "shrink": { "number_of_replicas": 0 }
        }
      },
      "delete": {
        "min_age": "60s",
        "actions": { "delete": {} }
      }
    }
  }
}

Creating the First Index

Assign the index to hot nodes, attach the log_ilm_policy, set rollover alias ilm_alias, 1 primary shard, 1 replica, and mark the alias as the write index:

PUT ilm_index-000001
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "log_ilm_policy",
    "index.lifecycle.rollover_alias": "ilm_alias",
    "index.routing.allocation.include.box_type":"hot"
  },
  "aliases": {
    "ilm_alias": { "is_write_index": true }
  }
}

Adding Documents

Post five documents to the alias:

POST ilm_alias/_doc
{
  "name":"cr7",
  "age":15
}

Observing the Lifecycle

Initially five docs are visible:

When the index reaches five docs, a rollover creates ilm_index-000002:

After 20 s the original index moves to warm nodes:

After 40 s it moves to cold nodes and replicas are reduced to 0:

After 60 s the index is deleted:

During warm or cold phases, attempts to write return a cluster_block_exception error.