Mastering Elasticsearch Index Design: From Sharding to Mapping

Introduction

Elasticsearch (ES) is a distributed RESTful search and analytics engine. Index design determines stability and resource efficiency for large‑scale log and analytics workloads.

Key Design Questions

What is ES index design?

Why plan ahead and what problems arise from poor design?

How to design indexes effectively?

How to configure shard and replica counts?

How to handle unpredictable index size (rollover)?

How to define mappings correctly?

Fundamental Design Considerations

Effective index design covers storage evaluation, shard sizing, read/write patterns, replica count, field types, and performance‑tuning parameters.

Common Pitfalls of Dynamic Index Creation

Fixed shard count : ES 7.x defaults to 1 primary shard; each shard can hold up to 2,147,483,519 documents. Changing the shard count later requires reindexing.

Write performance : Default 1 s refresh is unnecessary for write‑heavy logs; increasing refresh_interval (e.g., 60‑120 s) reduces indexing overhead.

Recovery performance : Adjust index.unassigned.node_left.delayed_timeout to delay shard allocation after a node leaves, reducing recovery load.

Immutable field types : Field types cannot be changed after creation; dynamic mapping may cause type conflicts, requiring full reindex.

Index Settings Design

Key settings include shard size (recommended 20‑40 GB), shard count, refresh_interval, replica count, and node allocation parameters.

1. Storage Evaluation

Assess business type, storage lifecycle, and expected shard configuration before creating an index.

Write‑heavy (log) use case : Use rollover indices to keep each index size manageable.

log_appcode-2021-10-01-000001
log_appcode-2021-10-01-000002
log_appcode-2021-10-01-000003

Read‑heavy use case : Increase shard count to improve query parallelism.

2. Rollover Example

PUT log_xxx-20211020-000001
{
  "aliases": {
    "log_xxx-alias": {
      "is_write_index": true
    }
  }
}

PUT log_xxx-alias/_bulk
{ "index": { "_id": 1 } }
{ "name": "zhangsan" }
{ "index": { "_id": 2 } }
{ "name": "lisi" }
...

POST log_xxx-alias/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_docs": 5,
    "max_size": "5gb"
  }
}

3. Replica Settings

PUT /index/_settings
{
  "settings": {
    "index.number_of_shards": "10",
    "index.number_of_replicas": "1"
  }
}

4. Index Recovery Settings

PUT /index/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}

5. Performance‑Optimization Settings

index.routing.allocation.total_shards_per_node

: limits total shards per node. index.refresh_interval: controls refresh frequency (default 1 s).

Example for a 10‑node cluster with 8 primary shards:

index.routing.allocation.total_shards_per_node: 2

For heavy log ingestion, set total_shards_per_node to 1 and refresh_interval to 120 s.

Mapping Design

Mappings define how documents and fields are stored and indexed. Two approaches:

Dynamic mapping : Fields are created automatically; suitable when the field set is unknown.

Static mapping : Explicit field definitions, analyzers, and storage options.

Mapping Design Process

Count business fields.

Determine field types and whether they need to be searchable.

Decide if fields require sorting, aggregation, or extra storage.

Key mapping parameters include type, index, norms, and ignore_above.

Complex Types

When nested data is required, ES provides:

Wide‑table pattern : Store related entities in a single index to avoid joins.

Nested type : Allows independent queries on objects within an array; updates are costly.

Parent‑child (join) type : Replaces nested for high‑frequency child updates, at the cost of extra memory.

Index Lifecycle Management (ILM) Example

A production log cluster uses a three‑stage lifecycle: apply (user‑requested storage extensions), maintain (metadata stored in PostgreSQL, approvals trigger updates), and clean (weekly Airflow job deletes old indices with a 3‑5 s pause per deletion).

Shard count is recalculated weekly using the _cat/indices and _cat/nodes APIs, then adjusted with a ~30 % buffer.

_cat/indices?format=json&pretty&bytes=b&h=index,pri,docs.count,pri.store.size

_cat/nodes?format=json&h=name,ip,node.role

Template Layers

Default templates : Provide common settings (e.g., 20 shards, 60 s refresh, 1 replica) for most log indices.

Custom templates : Tailored for specific business needs (additional fields, custom date formats).

{
  "order": 1,
  "index_patterns": ["log_*"],
  "settings": {
    "index": {
      "routing": {"allocation": {"total_shards_per_node": "1"}},
      "refresh_interval": "60s",
      "number_of_shards": "20",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {"type": "date"},
      "send_time": {"type": "long", "index": false},
      "app_code": {"type": "keyword", "index": false},
      "content": {"type": "text", "norms": false}
    }
  }
}

Best‑Practice Recommendations

Keep shard size between 20‑40 GB; each shard supports up to 2,147,483,519 documents.

Set shard count close to node count for optimal concurrency.

Use at least one replica in production; zero replicas only for reindexing.

Adjust total_shards_per_node based on the shard‑to‑node ratio.

Set refresh_interval to 1 s only when low latency is required; otherwise 30‑60 s is sufficient.

Prefer static mappings when the data model is known; use dynamic mapping or templates for uncertain fields.

Limit field count per index to roughly 1,000 and nesting depth to 3‑4 levels.

References

Elasticsearch mapping parameters: https://www.elastic.co/guide/en/elasticsearch/reference/7.7/mapping-params.html

Rollover API documentation: https://www.elastic.co/guide/en/elasticsearch/reference/7.7/indices-rollover-index.html

Index lifecycle management guide: https://www.elastic.co/guide/cn/elasticsearch/guide/current/index-management.html