Elasticsearch Advanced Guide: Core Concepts and Implementation Principles

Core Concepts

An index is a collection of documents that share similar characteristics and is identified by a lowercase name; the same name is used for indexing, searching, updating, and deleting documents. The term index can refer to both a noun (the logical collection) and a verb (the indexing operation).

A type was historically allowed in an index but has been removed since Elasticsearch 7.0; the default type is _doc.

A document is a single JSON‑encoded data unit that can be stored in an index.

A field corresponds to a column in a relational table and classifies document attributes.

Mapping defines field data types, analyzers, defaults, and indexing options; creating an optimal mapping improves performance.

PUT student
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 2
  },
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "sex":  { "type": "keyword" },
      "age":  { "type": "long" },
      "rank": { "type": "integer", "index": "false" }
    }
  }
}

Sharding and Replication

Each index can be split into multiple primary shards , each of which is a fully functional index that can be placed on any node. Sharding enables horizontal scaling of storage capacity and parallel processing for higher throughput.

Elasticsearch automatically creates a replica shard for each primary shard (unless disabled). Replicas provide high availability and allow search requests to be served in parallel, increasing read throughput.

The number of primary shards is fixed at index creation; the number of replicas can be changed dynamically.

Allocation

Shard allocation assigns primary or replica shards to specific nodes; the master node orchestrates this process.

Cluster Architecture

A running Elasticsearch instance is a node . A cluster consists of one or more nodes sharing the same cluster.name. When nodes join or leave, the cluster rebalances data automatically.

Any node can receive client requests; the coordinating node forwards the request to the appropriate shard(s) and aggregates the results.

Single‑Node Deployment Example

PUT users
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

Fault Tolerance

If a node that holds a primary shard fails, Elasticsearch elects a new master and promotes a replica shard to primary, ensuring the cluster remains operational.

Horizontal Scaling

Adding a third node triggers automatic shard reallocation, distributing load and increasing capacity. More replicas improve read throughput but do not increase write performance unless additional hardware is added.

Routing

Routing determines the target shard for a document using the formula shard = hash(routing) % number_of_primary_shards. By default, the document _id is used, but a custom value (e.g., user ID) can be supplied to keep related documents on the same shard.

Node Roles

All nodes can act as coordinating nodes, handling any request and forwarding it to the appropriate shard.

Write, Read, and Update Flows

Write flow: client → coordinating node → primary shard → replica shards; success is reported back after all replicas acknowledge.

Read flow: client → coordinating node → any shard (primary or replica) selected via round‑robin; the node returns the document to the client.

Update flow: read the document, modify the _source, re‑index the new version on the primary shard, then propagate to replicas. Updates are applied asynchronously to replicas.

Inverted Index

Elasticsearch uses an inverted index for fast full‑text search. Each unique term maps to a list of documents containing that term. The index is built by tokenizing document fields, applying analyzers, and storing term‑to‑document mappings.

Segment Management

Documents are first written to an in‑memory buffer, then periodically flushed to a new immutable segment. Segments are merged in the background to reduce the number of files, reclaim space from deleted documents, and improve search performance.

Translog and Durability

The transaction log (translog) records all operations before they are flushed to disk, enabling recovery after a crash. By default, the translog is fsynced every 5 seconds or after each write request. The index.translog.durability setting can be changed to async for higher throughput at the risk of data loss on crash.

Refresh and Flush

Each shard automatically performs a refresh roughly once per second, making newly indexed documents searchable. The refresh_interval can be increased (e.g., 30s) to improve indexing speed.

A flush writes in‑memory buffers to disk and creates a new commit point; it occurs automatically every 30 minutes or when the translog grows large.

Optimization Recommendations

Use SSDs and avoid remote storage (NFS/SMB) to reduce I/O latency.

Set an appropriate number of primary shards based on expected index size (e.g., keep shard size < 32 GB) and node count (shard count ≤ 3 × node count).

Adjust refresh_interval, translog.durability, and replica count to balance write latency and search freshness.

Allocate heap memory to ≤ 50 % of physical RAM and keep the JVM heap ≤ 32 GB for optimal Lucene caching.

Consider disabling replicas during bulk indexing and re‑enable them afterward.

Hardware and Memory Configuration

Elasticsearch stores data on local disks; configuring multiple path.data directories spreads data across disks. Allocate heap memory (e.g., -Xms31g -Xmx31g) but never exceed 50 % of physical RAM or 32 GB to avoid pointer overhead.