ElasticSearch Near Real-Time Search: Immutable Indexes, Segments, and Translog

1. Near Real-Time Search

1.1 Real-Time vs Near Real-Time

Real‑time search means a newly inserted document can be found immediately. Near real‑time is slightly slower, allowing a small delay after insertion.

1.2 Challenges of Near Real-Time

Achieving near real‑time on a single‑node system is difficult because it must ensure persistence and use caches to accelerate access. In a distributed system like ElasticSearch, it must also initialize full‑text search structures, making the problem more complex.

2. ElasticSearch Implementation

2.1 Immutable Data Structures

Concurrent programming struggles with mutable data. Functional programming solves this with immutable structures. ElasticSearch relies on Lucene’s immutable inverted index, which stores term statistics such as term frequency, document length, and position.

2.2 From Immutable to Mutable

When documents are added, Lucene builds a new immutable inverted index (segment). New data creates additional segments, which are merged at search time. Deletions and updates are handled via a logical delete structure (del) and by marking old documents, while new versions reside in fresh segments. Each segment is called a Segment, and the collection is managed as an Index.

In ElasticSearch, a database is called an Index; each Index can be divided into Shards, which are Lucene Indexes distributed across nodes and rebalanced under load.

The same concept appears in other structures like the Log‑Structured Merge (LSM) tree.

2.3 Distributed Data Storage

ElasticSearch shards data and routes it; each shard is a Lucene Index. Primary shards are synchronized to replica shards after operations.

2.4 Disk I/O Challenges

Segments are built in the filesystem cache and flushed to disk via fsync. ElasticSearch refreshes every second, creating a new segment from buffered documents, which provides near real‑time rather than true real‑time.

Frequent segment creation can lead to many small files, increasing file handles and search overhead. To mitigate this, a background merge thread combines small segments into larger ones, discarding deleted or outdated files and updating the commit point without affecting insert or search performance.

2.5 Ensuring Data Durability

To avoid data loss on process failure, ElasticSearch uses a write‑ahead log called translog. Every buffered document is also written to the translog. When a new segment is created, the buffer is cleared but the translog persists until the segment is flushed to disk.

On restart, ElasticSearch replays the translog after loading committed segments. By default, a full commit (flush) occurs every 30 minutes or when the translog grows large. The translog is synced to disk every 5 seconds, limiting potential data loss to a few seconds.

3. Further Learning

After gaining distributed systems experience, sections 2.3 and 2.5 can be skimmed, as they cover common concepts. Mastering ElasticSearch requires understanding both basic usage and the design principles explained in sections 2.1 and 2.2; deeper insight comes from reading the source code.