How Elasticsearch Indexes Documents: Step‑by‑Step Process

Document Index Steps

Single Document

When a client creates, indexes, or deletes a single document, the request is sent to a coordinating node (Node 1). The coordinating node hashes the document’s _id to determine the target shard (e.g., shard 0) and forwards the request to the primary shard’s node (Node 3). The primary shard processes the request and, upon success, forwards it in parallel to replica shards (Node 1 and Node 2). Once all replicas acknowledge success, the coordinating node returns a success response to the client.

Bulk Operations

For bulk requests, the client sends the bulk payload to a coordinating node (Node 1). The node creates a sub‑request for each target shard and forwards them concurrently to the nodes holding the primary shards. Each primary shard executes the operations sequentially: it writes the new document (or deletion) to its in‑memory buffer, then forwards the change to its replicas. After all replicas confirm success, the coordinating node aggregates the responses and returns them to the client.

Overall Indexing Flow

The coordinating node uses the document ID (or a custom routing value) to compute the shard via shard = hash(document_id) % (num_of_primary_shards). The request then follows the steps described above.

Write → Refresh → Flush → Merge Process

Write : A new document is first stored in the in‑memory buffer and recorded in the translog. At this point the document is not searchable.

Refresh : By default every second, the in‑memory buffer is written to a new segment stored in the filesystem cache, making the document searchable. The refresh interval can be changed via index.refresh_interval. Refresh incurs a performance cost, so a longer interval (e.g., 5 s) is often recommended.

Flush : When the translog grows large (default 512 MB) or after a periodic interval (default 30 min), the buffer is flushed to disk, creating a new commit point and clearing the translog.

Merge : Frequent refreshes create many small segments, which increase file‑handle usage, memory consumption, and search latency. A background merge process combines small segments into larger ones, eventually deleting the old segments after they are flushed to disk.

Write Mechanism in Elasticsearch

Elasticsearch builds on Lucene. Lucene’s IndexWriter provides three core methods:

public long addDocument();
public long updateDocuments();
public long deleteDocuments();

These methods handle single‑document operations but lack distributed support, real‑time visibility, durability, and partial updates. Elasticsearch adds:

Shard routing (primary + replicas) to achieve distribution.

Translog for durability: writes are first persisted to the translog before being flushed to disk.

Refresh to make in‑memory segments searchable.

Versioning, locks, and sequence IDs to guarantee atomicity and isolation.

Primary and Replica Workflow

When a write request reaches a primary shard, Elasticsearch writes the document to Lucene (in memory), then writes the same operation to the translog. After the primary succeeds, the request is sent to all replica shards, which repeat the Lucene and translog writes. The overall latency equals latency = primary_write + max(replica_writes). Replicas provide fault tolerance; if a replica fails, the master can remove it from the cluster.

Update Process

Because Lucene does not support partial updates, Elasticsearch implements them as follows:

Read the existing full document (or translog entry) to obtain version V1.

Merge the incoming partial fields with the full document, increment the version to V2, and lock the document.

Check for version conflicts; on conflict, retry from step 1.

Write the merged document to Lucene (delete‑then‑add) and update the version map.

Release the lock, completing the partial update.

Request Types

Elasticsearch supports four write request types: Index (create), Update, Delete, and Bulk. Since version 6.0, the single‑document operations are internally implemented via the bulk API.

Bulk Request Handling

The client node parses the bulk payload, optionally runs an ingest pipeline, auto‑creates the index if needed, sets routing, and groups individual write actions by target shard into BulkShardRequest objects. Each BulkShardRequest is sent to the primary shard, which processes the actions (translating updates to index/delete, parsing docs, updating mappings, obtaining sequence IDs and versions, adding docs to Lucene, writing the translog, optionally flushing, and finally forwarding the resulting index/delete actions to replicas). Replicas execute the simplified actions, write their translog, and acknowledge the primary.

Key Properties Revisited

Reliability : Achieved through primary‑replica replication and the translog.

Consistency : Elasticsearch provides eventual consistency; replicas may lag behind the primary until they apply the same operations.

Atomicity : Add and delete operations are atomic; versioning and locks ensure atomicity for partial updates.

Isolation : Enforced via version numbers and per‑document locks.

Real‑time Visibility : Refresh (default 1 s) makes recent writes searchable; Get‑by‑ID can read directly from the translog for near‑real‑time access.

Performance : Designed to minimize latency—replicas can return before all have responded, segments are served from memory, and version maps reduce hot‑spot I/O.