Why Elasticsearch Is Called Near‑Real‑Time and How It Works Under the Hood

Interview Scenarios

Scenario 1: The interviewer asks, “What is your understanding of Elasticsearch?” The candidate replies that ES has excellent performance and that all order data is synchronized to ES, with all queries routed through it. When asked whether high‑real‑time ID queries also use ES, the candidate hesitates, exposing a gap in understanding the “near‑real‑time” claim.

Scenario 2: The candidate describes ES as a Java‑based search engine built on Lucene, distributed, scalable, and capable of handling multi‑dimensional search. When prompted about what ES is not suitable for, the answer is vague.

Scenario 3: The interviewer mentions a system with a peak QPS of 3000 and asks how to handle a ten‑fold increase. The candidate suggests adding more ES nodes, but fails to analyze request types, illustrating a common interview pitfall.

Application Scenarios

In internet and e‑commerce domains, ES is often used to complement MySQL’s shortcomings, especially for full‑text search and complex queries. Building indexes for every search field in MySQL would degrade write performance, and sharding further complicates queries. Approximately 80% of ES adoption in these fields addresses these problems.

Inverted Index

Elasticsearch relies on inverted indexes, which consist of a term dictionary and posting lists. Unlike forward indexes that map document IDs to terms, an inverted index maps terms to document IDs, enabling fast retrieval of matching documents.

The posting list stores:

Document ID (primary key)

Term frequency (how often the term appears in the document, used for scoring)

Position (token order, used for phrase queries)

Offset (character offset of the term)

By default, each field in an ES JSON document has its own inverted index, which is why ES outperforms MySQL on complex queries.

Tokenizer and Analyzer

Analyzers convert raw text into tokens. An analyzer consists of three components executed in order:

Character Filters : preprocess the raw text (e.g., HTML stripping).

Tokenizer : split the text into tokens; the default is the Standard Tokenizer.

Token Filters : post‑process tokens (lower‑casing, stop‑word removal, synonym expansion, etc.).

These steps ensure that duplicate terms are removed and the token stream is sorted for efficient search.

Near‑Real‑Time Search

ES achieves near‑real‑time visibility by writing new segments to the file‑system cache first, then flushing to disk later. Lucene allows a newly written segment to be searchable without a full commit, reducing latency from minutes to seconds. By default, each shard refreshes automatically every second, making newly indexed documents searchable within one second.

Refresh can be controlled via the refresh_interval setting:

POST /_refresh

POST /blogs/_refresh

PUT my_index/_settings
{
  "index": {
    "refresh_interval": "30s"
  }
}

Setting

refresh_interval" to "-1" disables automatic refresh, useful when bulk‑loading large indexes.</p>
<h2>Search Types</h2>
<p>Elasticsearch supports four search types:</p>
<ul>
<li><strong>query_and_fetch</strong> (local): each shard returns full documents and scores; fast but may return more documents than requested and ranking is shard‑local.</li>
<li><strong>query_then_fetch</strong> (default, local): shards return only document IDs and scores; ES re‑ranks globally and fetches the exact number of requested documents.</li>
<li><strong>DFS_query_and_fetch</strong> (global): adds a distributed‑frequency‑statistics (DFS) step before querying to improve scoring accuracy; slower and may return excess documents.</li>
<li><strong>DFS_query_then_fetch</strong> (global): combines DFS with the query‑then‑fetch flow; most accurate but has the highest latency.</li>
</ul>
<h2>Query vs. Filter</h2>
<p>Search operations consist of <strong>queries</strong> (full‑text, relevance‑scored) and <strong>filters</strong> (binary inclusion/exclusion). Queries compute a <code>_score

for each document, are not cached, and are slower. Filters simply decide whether a document matches, can be cached, and are faster.

Example combining both:

GET /_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "Search" }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [
        { "term": { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}

Typical use cases:

Full‑text search where relevance ranking matters.

Exact matches, range queries, or boolean conditions where performance is critical.

Conclusion

Understanding the concepts above—near‑real‑time indexing, inverted indexes, analyzers, search types, and the query/filter distinction—provides a solid foundation for using Elasticsearch effectively and making informed technology choices based on business requirements.