Skip to content
Elasticsearch Core Internals: Inverted Index, Cluster, Write/Query, IK, Deep Pagination & Optimization
Databases 36 min read

Elasticsearch Core Internals: Inverted Index, Cluster, Write/Query, IK, Deep Pagination & Optimization

A comprehensive technical guide covering Elasticsearch inverted index internals (FST, skip lists), cluster architecture with node roles and shard/replica design, write path (refresh, translog, flush), query execution (Query Then Fetch), IK Chinese analyzer with hot reload, deep pagination solutions (search_after vs scroll), and production index optimization across design, write, query, and operations dimensions.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Inverted Index Internals

Elasticsearch's inverted index is the core data structure enabling full-text search. It maps terms to document IDs (postings lists), opposite of forward indexes. The index comprises three key structures:

Term Dictionary : Sorted unique terms supporting prefix/fuzzy queries.

Posting List : Stores document IDs plus term frequency (TF), positions, and offsets for scoring, highlighting, and phrase queries.

Term Index (FST) : A Finite State Transducer acting as an "index of the index," mapping term prefixes to disk locations. FST compresses memory usage to ~1/10 of a hash map, reducing lookup from O(n) to O(log n).

Segments are immutable once written, enabling lock-free concurrent reads. Near-real-time search uses a 1-second default refresh to make new segments searchable via OS cache. Doc Values (columnar storage) handle sorting/aggregations efficiently.

Skip Lists & FST Compression

Posting lists use skip lists to accelerate AND/OR merges. FST compression allows millions of terms to occupy only a few MB of heap. Segment merging periodically consolidates small segments to reduce I/O.

Cluster Architecture: Node Roles, Shards & Replicas

Nodes are JVM processes with distinct roles:

Master : Cluster metadata, shard allocation, node membership. Config: node.roles: [master]. Production: deploy 3 dedicated odd-numbered nodes to prevent split-brain.

Data : Store shards, execute CRUD/search/aggregations. Config: node.roles: [data]. Production: use SSD + large heap; separate hot/warm tiers.

Coordinating : Route requests, merge results, offload aggregation memory. Config: node.roles: []. Production: deploy dedicated under high load.

Ingest : Pre-process documents via pipelines (Grok, enrich). Config: node.roles: [ingest]. Production: separate for complex transformations.

Critical : Production must separate roles; mixing Master+Data risks GC-induced master instability and split-brain.

Shards & Replicas

A shard is a Lucene index. Primary shards (fixed at creation) are the write entry point; replicas provide HA and read scaling. Routing formula: shard_num = hash(_routing) % num_primary_shards. Changing primary count breaks routing, requiring reindex.

Shard size: 10–50 GB (author suggests 20–40 GB).

Replicas: 1–2 typical; more hurts write throughput and disk.

Primary and replica never on same node.

Write goes to primary then syncs to replicas; wait_for_active_shards controls durability.

Reads round-robin across primary/replicas for linear read scaling.

Example 3-node cluster with 3 primaries + 1 replica each:

Node 1: P0, R1
Node 2: P1, R2
Node 3: P2, R0

Kill Node 1 → R0 promotes to primary instantly.

Write Path: Refresh, Translog, Flush

Write flow:

memory buffer + translog → refresh → in-memory segment (searchable) → flush → segment on disk + clear translog

.

refresh : Trigger: periodic/manual. Purpose: make data searchable (NRT). Durability: no (OS cache only). Performance: medium (CPU for inversion). Default: 1s.

translog : Trigger: every write/periodic fsync. Purpose: prevent data loss on crash. Durability: yes (fsync per request by default). Performance: low (sequential write). Default: 5s or request sync.

flush : Trigger: translog full/30min/manual. Purpose: full persistence to disk. Durability: yes (segment + translog cleared). Performance: high (heavy I/O). Default: 30min or 512MB.

Common misconceptions : refresh ≠ disk persistence; translog written before flush, not after; flush ≠ refresh.

Optimization Tips

Non-real-time (logs): increase refresh_interval to 30–60s for 2–5× write throughput.

High write, lower durability: index.translog.durability: async with sync_interval: 30s.

Avoid manual flush; tune index.translog.flush_threshold_size.

Query Execution: Query Then Fetch

Default search type splits work into two phases:

Query (scatter) : Coordinator broadcasts to all relevant shards. Each shard runs the query locally, builds a priority queue, and returns only doc_id + sort fields (e.g., _score) for from+size docs. Coordinator merges and globally sorts, then truncates to final top N.

Fetch (gather) : Coordinator issues Multi-Get requests for the selected doc IDs to retrieve full _source, highlights, stored fields.

Deep pagination root cause : For from=9990, size=10, each shard returns 10,000 docs; with 10 shards, coordinator sorts 100,000 docs → memory/CPU explosion. Hence index.max_result_window defaults to 10,000.

Pagination Solutions

from+size : Max results 10,000. Jump pages: yes. Real-time: yes. Performance: fast shallow, terrible deep. Memory: high deep. Use case: web pagination (first ~100 pages).

scroll : Max results unlimited. Jump pages: no. Real-time: no (snapshot). Performance: fast full scan. Memory: high (context). Use case: bulk export, offline analysis.

search_after : Max results unlimited. Jump pages: no. Real-time: yes. Performance: stable. Memory: very low. Use case: infinite scroll, mobile feeds. search_after uses the last hit's sort values (must include a unique tie-breaker like _id) as the next query's cursor. Each shard only fetches size docs matching sort_field > last_value, so coordinator merge size stays constant ( shards * size).

GET /order/_search
{
  "size": 10,
  "sort": [ { "order_id": "asc" } ],
  "search_after": [ 1024 ]
}

Analyzers & IK Chinese Tokenizer

Analyzer pipeline: Character Filters (HTML strip, emoji replace) → Tokenizer (split into tokens) → Token Filters (lowercase, stopwords, synonyms, stemming). Index-time and search-time analyzers must match or searches miss.

IK provides two modes: ik_smart: coarse-grained (e.g., ["我", "爱", "北京", "天安门"]) ik_max_word: fine-grained exhaustive (e.g., ["我", "爱", "北京", "天安门", "天安", "门"])

Typical mapping: index with ik_max_word, search with ik_smart for precision/recall balance.

Hot Reload via Remote Dictionary

Configure IKAnalyzer.cfg.xml with HTTP endpoints:

<entry key="remote_ext_dict">http://host:port/ik/words</entry>
<entry key="remote_ext_stopwords">http://host:port/ik/stopwords</entry>

IK polls every 60s; if Last-Modified changes, it reloads without restart. Backend can serve from MySQL, enabling ops to add terms via admin UI.

Common Pitfalls

Mismatched index/search analyzers → missing hits.

Missing custom dictionary → brand names split incorrectly ("小米手机" → "小","米","手机").

Wrong stopwords → important terms dropped.

Remote dict missing Content-Type: text/plain or Last-Modified header.

IK version ≠ ES version → plugin fails to load.

Debug with _analyze API and _termvectors to inspect actual indexed tokens.

Index Optimization Checklist

Design Phase (80% impact)

Shard count: target 20–40 GB/shard; shards = nodes × 1.5–3. Example: 3 nodes, 100 GB → 6 shards.

Replicas: 0 during bulk load, then 1–2 for HA.

Mappings: disable _all (ES7+ default); use keyword for exact match, index: false for non-searchable; text+keyword multi-fields for dual needs.

ILM: time-based indices (daily logs) with hot/warm/cold/delete phases.

Dynamic mapping: dynamic: strict to prevent field explosion.

Numeric types: smallest sufficient (byte/short/integer); scaled_float for prices.

Large fields: enabled: false on _source or exclude from indexing.

Classic bug: storing enum status as text then querying with term fails because text is analyzed. Fix: use keyword.

Write Optimization

Bulk API: 5–15 MB batches, ≤1000 docs.

Increase refresh_interval to 30–60s for write-heavy loads.

Merge policy: LogByteSizeMergePolicy with max_merge_at_once=10, segments_per_tier=10.

Translog: durability: async, sync_interval: 30s for higher throughput.

Query Optimization

Avoid match_all on large indices; always filter.

Routing: co-locate related docs (e.g., by merchant_id) to query single shard.

Pagination: from+size ≤10k; search_after for deep; scroll for export.

Prefer filter over query clauses: no scoring, cached, 3–10× faster.

Operations

Role separation: dedicated Master, Data, Coordinating, Ingest.

Heap: 50% RAM, max 31 GB (compressed oops limit).

SSD mandatory; disk watermark at 85%.

Monitor: shard health, merge rate, GC, disk, latency.

Scenario Patterns

Log ingestion : Throughput over real-time. Key settings: refresh_interval=30s, async translog, 10 MB bulk.

E-commerce search : Low latency, routing. Key settings: routing by user/product, filter cache, no script scoring.

Disk pressure : Compression, cleanup. Key settings: codec: best_compression, force merge read-only, rollover delete.

Frequent GC : Heap pressure. Key settings: shard <50 GB, heap <30 GB, segment count <500.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

ElasticsearchQuery OptimizationDeep PaginationInverted IndexIndex OptimizationWrite PathCluster ArchitectureIK Analyzer
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.