How Elasticsearch Writes Data: Buffers, Translog, Refresh, and Merge Explained

2. Elasticsearch Overall Structure

Elasticsearch clusters consist of multiple server nodes; each index is divided into shards, and each shard has several replicas. One replica is the primary shard responsible for writes, while other replicas are read‑only copies that stay synchronized with the primary.

When a write request arrives, Elasticsearch routes it to the target shard's primary replica.

Each shard is a Lucene index containing multiple segment files and a commit point file. Segment files store individual documents, and the commit point records which segment files exist.

3. What If Directly Writing Segments Is Inefficient?

Writing directly to disk for every operation would result in low I/O efficiency.

Therefore, Elasticsearch uses an in‑memory buffer to temporarily hold incoming data.

Because memory is volatile, Elasticsearch also writes every operation to a transaction log (Translog), similar to MySQL's binlog, enabling recovery after a crash.

Writes are not flushed to disk immediately; they are buffered and written every few seconds, so up to a few seconds of data may be lost unless the flush interval is adjusted.

Thus, after reaching the primary shard, data first enters the buffer and the operation is recorded in the Translog.

4. How Does Buffer Data Write to Segment Files?

Every second Elasticsearch performs a refresh operation, creating a new segment file, moving buffered data into that segment, and clearing the buffer.

Data written to a segment becomes part of Lucene, building an inverted index that can be searched.

5. How to Further Improve Segment Write Efficiency?

Although data is written to segment files, it may still reside in the operating system's page cache; the actual physical write occurs later unless the application forces an fsync.

Elasticsearch deliberately avoids fsync for performance, relying on the Translog to guarantee durability.

Even if the segment file hasn't been physically persisted, as soon as it reaches the OS cache it is searchable.

6. What to Do When the Translog Grows Too Large?

The Translog expands as more data is written and must be cleaned up.

Cleanup is triggered when either of two conditions is met: the file size exceeds a configured threshold, or 30 minutes have passed.

When triggered, Elasticsearch performs a flush sequence:

Execute a refresh.

Force all uncommitted segment data to be flushed to disk.

Create a commit point that records all current segments.

Clear the Translog because the segment data is now safely persisted.

7. What If There Are Too Many Segments?

Frequent refreshes can generate many small segment files, degrading search performance.

Elasticsearch runs a background merge process that combines small segments into larger ones and updates the commit point accordingly.

The merge also removes documents that were marked for deletion; deletions are first recorded in a .del file and only physically removed during merge.

8. Summary

During a write, the client request is routed to the primary shard's server node; the document is placed in the buffer and logged in the Translog.

Every second a refresh creates a segment file from the buffer and clears the buffer.

The segment data is cached in memory and later flushed to physical storage during a flush operation.

When the Translog grows large or every 30 minutes, a flush forces all segment data to disk, creates a commit point, and clears the Translog.

The merge process continuously consolidates small segments and permanently removes deleted documents.