Master ElasticSearch: From Installation to Advanced Index and Memory Optimization

1. Introduction

ElasticSearch (ES) is a distributed, RESTful search and analytics engine built on Lucene, offering lightweight deployment, schema‑free JSON indexing, multi‑index support and easy clustering. It has been adopted by GitHub, SoundCloud, Baidu and many others for large‑scale search and analytics.

2. Core Concepts

Cluster and Node – A cluster is a group of nodes that together provide the search service; each node runs an ES instance.

Index – Logical storage similar to a database.

Shards – Primary pieces of an index distributed across nodes.

Replicas – Copies of shards for fault‑tolerance and load‑balancing.

Recovery – Data redistribution when nodes join or leave.

Gateway – Snapshot storage mechanism for index data.

Discovery.zen – Automatic node discovery via broadcast and multicast.

Transport – Internal communication using TCP (default) and HTTP (JSON).

3. Installation & Deployment

Download elasticsearch-1.6.0.tar.gz, extract it, and edit config/elasticsearch.yml with minimal settings:

cluster.name: elasticsearch
node.name: "node1"
node.data: true
index.number_of_shards: 5
index.number_of_replicas: 1
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/log
index.cache.field.max_size: 500000
index.cache.field.expire: 5m

Start ES with bin/elasticsearch -d -Xms512m -Xmx512m and verify the service by opening http://ip:9200/; a HTTP 200 response indicates a successful start.

4. Data Indexing (Python example)

Install the official Python client: pip install elasticsearch Create an index and bulk‑load documents using the bulk API. Example screenshots illustrate the process:

5. Index Optimization

Key settings to speed up indexing:

Increase index.translog.flush_threshold_ops (default 5000) or set to -1 to disable frequent translog flushes.

Adjust index.refresh_interval (default 120s) or disable during bulk loading, then manually refresh when needed.

Set number_of_replicas to 0 while loading data, and restore the desired replica count after indexing completes.

6. Memory Optimization

ES runs on the JVM; heap should not exceed half of the physical RAM and stay below 32 GB. Important memory consumers include:

Segment memory – In‑memory term dictionary and segment metadata that cannot be garbage‑collected; more segments mean higher heap usage.

Filter cache – Caches filter results, defaulting to 10 % of heap.

Field data cache – Used for sorting and aggregations; prefer doc values to avoid heap pressure.

Bulk queue , indexing buffer , cluster state buffer – Each has sensible defaults; avoid excessive tuning that can increase heap consumption.

Monitor segment memory via the CAT API (e.g., GET /_cat/segments) and reduce it by deleting unused indices, closing indices, or force‑merging segments (force merge API).

7. Practical Recommendations

Run on JDK 1.7+ (prefer Oracle JDK 1.8) for stability.

Keep shard size ≤ 10 GB; tune the number of shards and replicas according to hardware and data volume.

Use doc values instead of field data cache for large aggregations.

Limit query size and from parameters; use the scroll API for deep pagination.

Continuously monitor heap, segment memory, and cache usage, and adjust configurations based on observed metrics.