Mastering Elasticsearch Index Lifecycle Management in Kibana

Introduction

After a few days of log collection, the ELK cluster’s index count grew, causing memory, CPU, and I/O metrics on the servers to rise; to address this, index lifecycle management should be implemented while balancing performance and user experience.

Kibana Index Configuration

1. Manage Indices

Click Settings –> Elasticsearch Index Management to view all indices generated by ELK.

Configure Kibana Index Patterns

Configure Index Lifecycle

Click Settings –> Elasticsearch Index Lifecycle Policies to configure policies that manage the index lifecycle.

The policy documentation is available at https://www.elastic.co/guide/en/elasticsearch/reference/7.1/index-lifecycle-management.html. Setting up the index lifecycle took several days, and the following configuration was used:

First, create Index Lifecycle Policies covering the four official phases; not every phase is required. The configuration used is illustrated below:

Hot Phase

Warm Phase

The index is moved to this phase three days after creation, and a force merge to three segments is configured.

Cold Phase

The index is moved to this phase seven days after creation and is frozen.

Delete Phase

The index is deleted fifteen days after creation.

After configuring the lifecycle policy, a template must be created to take over existing indices and apply the policy to the template, ensuring every newly created index follows the policy. Using a "logstash-*" prefix is the simplest approach, but a custom template can be defined as follows:

PUT /_template/my_template</code>
<code>{</code>
<code>  "order": 0,</code>
<code>  "index_patterns": ["prod-*", "stage-*", "dev-*"] ,</code>
<code>  "settings": {</code>
<code>    "index": {</code>
<code>      "number_of_shards": "1",</code>
<code>      "refresh_interval": "5s"</code>
<code>    }</code>
<code>  },</code>
<code>  "mappings": {</code>
<code>    "dynamic_templates": [</code>
<code>      {</code>
<code>        "message_field": {</code>
<code>          "path_match": "message",</code>
<code>          "mapping": { "norms": false, "type": "text" },</code>
<code>          "match_mapping_type": "string"</code>
<code>        }</code>
<code>      },</code>
<code>      {</code>
<code>        "string_fields": {</code>
<code>          "mapping": { "norms": false, "type": "text", "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } } },</code>
<code>          "match_mapping_type": "string",</code>
<code>          "match": "*"</code>
<code>        }</code>
<code>      }</code>
<code>    ],</code>
<code>    "properties": {</code>
<code>      "@timestamp": { "type": "date" },</code>
<code>      "geoip": { "dynamic": true, "properties": { "ip": { "type": "ip" }, "latitude": { "type": "half_float" }, "location": { "type": "geo_point" }, "longitude": { "type": "half_float" } } },</code>
<code>      "@version": { "type": "keyword" }</code>
<code>    }</code>
<code>  },</code>
<code>  "aliases": {}</code>
<code>}

The above template takes over indices starting with prod-*, stage-*, or dev-*, and applies the lifecycle policy to them.

Now you can view the current lifecycle status of each index in Index Management.

Conclusion

While this process achieves the expected results, many concepts such as shards and segments still require deeper understanding to optimize allocation and performance.