Essential Elasticsearch Tuning Tips for Performance and Stability

1. Configuration File Tuning

Memory Lock : Set bootstrap.memory_lock: true to prevent the OS from swapping JVM memory.

Discovery Settings : Use unicast discovery ( discovery.zen.ping.unicast.hosts) and avoid multicast in production. Adjust discovery.zen.ping_timeout, discovery.zen.join_timeout, and discovery.zen.minimum_master_nodes to control master election timing.

Fault Detection : Configure discovery.zen.fd.ping_interval (default 1s), discovery.zen.fd.ping_timeout (default 30s), and discovery.zen.fd.ping_retries (default 3) to monitor node liveness.

Queue Size : Increase queue size only when GET /_cat/thread_pool shows persistent queue rejections; otherwise, larger queues waste memory.

Memory Breakers : Tune circuit breakers such as indices.breaker.total.limit (default 70% of JVM heap), indices.breaker.request.limit (10% of heap), and indices.breaker.fielddata.limit (10% of heap) to avoid OOM.

Cache Size : Set indices.queries.cache.size (e.g., 5%) to limit query cache memory.

2. System‑Level Tuning

JDK Version : Use the JDK version recommended by the official Elasticsearch documentation.

JVM Heap : Set -Xms and -Xmx to the same value, typically slightly less than half of system RAM (if RAM < 64 GB). Keep heap size ≤ 32 GB to avoid pointer‑compression overhead.

Swap : Disable swap (e.g., swapoff -a) to prevent performance degradation.

File Descriptors : Increase the limit (e.g., ulimit -n 65536) because Elasticsearch and Lucene open many files and sockets.

Memory‑Mapped Files : Set vm.max_map_count=262144 (persist via /etc/sysctl.conf) to provide enough virtual memory for mmap.

Disk I/O Scheduler : On SSDs, use deadline or noop scheduler ( echo noop > /sys/block/sd/queue/scheduler) instead of the default CFQ.

Disk Mount Options : Mount data volumes with noatime,data=writeback,barrier=0,nobh to reduce latency and avoid journaling overhead.

RAID : Prefer RAID 0 for raw I/O performance; avoid remote mounts (NFS/SMB) that add latency.

3. Elasticsearch Usage Tuning

Hot Threads : Run GET /_nodes/hot_threads?interval=30s to identify resource‑heavy threads (e.g., bulk, search, merge).

Pending Tasks : Use GET /_cluster/pending_tasks to spot metadata‑change bottlenecks such as excessive shard recovery or mapping updates.

Field Storage : Choose appropriate field types— doc_values for most keyword fields, disable fielddata when not needed, disable _source if updates/reindex are unnecessary, turn off _all in newer versions, and disable norms for log data.

Translog : Set index.translog.durability: async for async fsync, adjust index.translog.sync_interval (e.g., 5s), and limit size with index.translog.flush_threshold_size (default 512 MB).

Refresh Interval : Default 1 s; increase or set -1 to disable for high‑throughput indexing.

Dynamic Mapping : Disable or set to false / strict to avoid uncontrolled metadata growth; understand the three options (true, false, strict).

Bulk Indexing : Send batches of 5–15 MB (physical size) rather than counting documents; monitor with iostat, top, ps. Reduce concurrency if EsRejectedExecutionException occurs.

Index and Shard Management : Use shrink and rollover APIs, keep shard size ≤ 50 GB for logs or ≤ 20 GB for business data, and name indices by time (e.g., test-YYYYMMDD).

Segment Merge : Limit merge threads with index.merge.scheduler.max_thread_count; on HDDs set low values, SSDs use default.

Auto‑Generated IDs : Prefer Elasticsearch‑generated IDs to avoid costly existence checks on custom IDs.

Routing : Specify a routing key on write to limit queries to a single shard, reducing merge and coordination overhead.

Aliases : Serve traffic through index aliases to allow seamless reindexing or index swaps.

Avoid Wide Tables : Limit total fields per index ( index.mapping.total_fields.limit, default 1000) to prevent mapping explosion.

Avoid Sparse Indexes : Sparse fields increase Lucene’s delta‑encoding size, inflating disk usage and slowing queries.

For detailed parameter references, see the official Elasticsearch documentation links included in the original article.