Elasticsearch Index and Search Optimization Guide

Overview

The article begins with a high‑level diagram of an Elasticsearch cluster, focusing on the index and search modules built on Lucene and explains how configuration changes affect cluster behavior.

Index Optimization

It describes the flow of a document write request and identifies four key components involved in posting‑list building: heap buffer, OS cache, transaction log, and disk. The optimization points target these components. mapping: disable unnecessary features, keep schema minimal, use field("index", "no") for non‑searchable fields.

Disable _all and set index.query.default_field to a specific schema. indices.memory: increase the indexing buffer size (e.g., indices.memory.index_buffer_size) to reduce refresh frequency. index.translog.durability: choose synchronous or asynchronous flush; tune index.translog.flush_threshold_opts and index.translog.flush_threshold_size. segment merge: limit merge speed with indices/index.store.throttle.max_bytes_per_sec and thread count with index.merge.scheduler.max_thread_count. refresh_interval: adjust to balance real‑time search and cache utilization. number_of_replicas: start with 0 during initial bulk load, enable later for high availability and query parallelism. number_of_shards: choose shard count based on data volume (20‑30 GB per shard) and node count (≈1.5‑3 × nodes).

Use auto‑generated document IDs to avoid existence checks on each write.

Separate master, data, ingest, and coordinating nodes for better fault isolation.

Prefer SSDs and RAID 0 for disk storage to reduce I/O latency.

Control index thread pool sizes (e.g., _cluster.threadpool.index.queue_size) and bulk request size.

Search Optimization

The search process is a two‑phase query‑then‑fetch workflow. Optimizations include routing, shard/replica settings, filter clauses, appropriate field types, and avoiding heavy nested or parent‑child queries.

Set routing to co‑locate related documents on the same shard, reducing the number of shards queried.

Adjust number_of_shards and number_of_replicas similarly to indexing.

Prefer filter clauses when scoring is not required.

Choose the smallest suitable data type (keyword, byte, short, integer, long, float, double).

Use nested instead of parent‑child where possible.

Limit max_num_segments to keep segment count low and improve query speed.

Increase filesystem cache so more index segments reside in memory.

Simplify document models; push heavy aggregations to client‑side code (Java/Spark).

Pre‑index range data as separate fields to enable term queries.

Separate hot and warm data at the node level using node.attr.box_type and index.routing.allocation.require.box_type.

Avoid deep pagination; use scroll_api or search_after for large result sets.

System Configuration

Key JVM and OS settings include heap size, GC algorithm (CMS or G1), thread limits, disabling swap, file descriptor limits, and virtual memory settings.

Stress Test

Performance can be measured with esrally, defining tracks, cars, races, tournaments, and pipelines. Example command:

esrally --distribution-version=5.0.0 --track=geopoint --challenge=append-fast-with-conflicts --car="16gheap"

Monitoring

Cluster health is observed via Elasticsearch plugins and the _cat API, using tools such as Kibana, Marvel, Kopf/Cerebro, and Head. Important metrics include search/index rate and latency, index size, segment count, JVM heap usage, CPU utilization, and system load.

Notes

Because Elasticsearch evolves quickly, always consult the documentation for the specific version you deploy and verify configuration compatibility.

References

How to Maximize Elasticsearch Indexing Performance

Anatomy of an Elasticsearch Cluster

Tune for indexing speed

Tune for search speed

Elasticsearch: The Definitive Guide