Why Elasticsearch Query Latency Spikes Occur and How to Diagnose Them

When a business is sensitive to query latency, occasional spikes in Elasticsearch response time can be hard to reproduce and diagnose because the problematic moment has already passed. The article enumerates typical factors that cause such spikes and presents concrete methods to locate and resolve them.

GC Impact

Long garbage‑collection pauses on any shard node can extend the overall query time. To pinpoint GC‑related delays, check node GC metrics in Kibana or review GC logs.

Typical remediation includes adjusting JVM heap size, reducing off‑heap memory pressure from open indices, FST structures, aggregations, Netty buffers, and caches. Memory usage can be inspected via the REST API:

curl -sXGET "http://localhost:9200/_cat/nodes?h=name,port,segments.memory,segments.index_writer_memory,segments.version_map_memory,segments.fixed_bitset_memory,fielddata.memory_size,query_cache.memory_size,request_cache.memory_size&v"

Off‑heap FST can lower heap usage, but may trigger occasional I/O when the off‑heap data is evicted.

System Cache Misses

Elasticsearch relies on the OS page cache for many file reads. When the cache is evicted, disk I/O occurs, noticeably increasing latency. Understanding which files a query reads helps assess the I/O cost.

Which files are read in real time during the query?

How many I/O operations occur, how many bytes are read, and how long does it take?

Files Accessed by Query Type

Different query types touch different Lucene files. The diagram below summarizes typical file access patterns.

Only query, no fetch (size=0) reads the tim file for term or match queries because the posting list is not needed.

_search?size=0
{
  "query": {
    "match": {"name": {"query": "Farasi"}}
  }
}

When a fetch phase is added (size>0), additional files such as fdt and fdx are read to retrieve stored fields.

_search?size=1
{
  "query": {"term": {"country_code.raw": {"value": "CO"}}}
}

Match queries also need the norms file ( nvd) because scoring requires norm values.

_search?size=10
{
  "query": {"match": {"name": {"query": "Farasi"}}}
}

Numeric range queries use BKD‑tree point values, reading the dim file.

_search?size=0
{
  "query": {"range": {"geonameid": {"gte": 3682501, "lte": 3682504}}}
}

Aggregations (both metric and bucket) read the dvd file when size=0.

_search?size=0
{
  "aggs": {"name": {"terms": {"field": "name.raw"}}}
}

The GET API for a single document first performs a Lucene lookup (termsEnum.seekExact) that reads the FST and tim, then fetches fdx / fdt and dvd for metadata.

_doc/IrOMznAB5onF36XmwY4W

Measuring I/O Impact

To observe actual I/O during queries, a SystemTap script hooks read and pread syscalls and logs byte counts and latency. Tests are run both with page cache cleared (using vmtouch and _cache/clear) and with cache retained.

Results show that most queries perform few read calls and small data volumes, but two scenarios can cause significant I/O:

Aggregations, where I/O scales with the amount of data being aggregated.

Numeric range queries, where I/O depends on the size of the hit set.

Additional cases with moderate I/O include multi‑condition queries (many postings list reads) and deep pagination (large document fetches).

Search Queue Backlog

Excessive concurrent query traffic can saturate the search thread pool, causing requests to queue and increasing latency. Monitoring this requires custom metrics, as Kibana does not expose thread‑pool queue length by default.

Mitigation strategies include limiting client concurrency and configuring max_concurrent_shard_requests to bound per‑node shard request parallelism.

Conclusion

Elasticsearch’s underlying Lucene engine is not optimized for ultra‑low latency; query spikes are mainly driven by GC pauses and I/O variability. Proper JVM memory planning, SSD or RAM‑disk usage, and sufficient page‑cache allocation are effective ways to reduce latency jitter.