Elasticsearch Core Internals: Inverted Index, Cluster, Write/Query, IK, Deep Pagination & Optimization
A comprehensive technical guide covering Elasticsearch inverted index internals (FST, skip lists), cluster architecture with node roles and shard/replica design, write path (refresh, translog, flush), query execution (Query Then Fetch), IK Chinese analyzer with hot reload, deep pagination solutions (search_after vs scroll), and production index optimization across design, write, query, and operations dimensions.
Inverted Index Internals
Elasticsearch's inverted index is the core data structure enabling full-text search. It maps terms to document IDs (postings lists), opposite of forward indexes. The index comprises three key structures:
Term Dictionary : Sorted unique terms supporting prefix/fuzzy queries.
Posting List : Stores document IDs plus term frequency (TF), positions, and offsets for scoring, highlighting, and phrase queries.
Term Index (FST) : A Finite State Transducer acting as an "index of the index," mapping term prefixes to disk locations. FST compresses memory usage to ~1/10 of a hash map, reducing lookup from O(n) to O(log n).
Segments are immutable once written, enabling lock-free concurrent reads. Near-real-time search uses a 1-second default refresh to make new segments searchable via OS cache. Doc Values (columnar storage) handle sorting/aggregations efficiently.
Skip Lists & FST Compression
Posting lists use skip lists to accelerate AND/OR merges. FST compression allows millions of terms to occupy only a few MB of heap. Segment merging periodically consolidates small segments to reduce I/O.
Cluster Architecture: Node Roles, Shards & Replicas
Nodes are JVM processes with distinct roles:
Master : Cluster metadata, shard allocation, node membership. Config: node.roles: [master]. Production: deploy 3 dedicated odd-numbered nodes to prevent split-brain.
Data : Store shards, execute CRUD/search/aggregations. Config: node.roles: [data]. Production: use SSD + large heap; separate hot/warm tiers.
Coordinating : Route requests, merge results, offload aggregation memory. Config: node.roles: []. Production: deploy dedicated under high load.
Ingest : Pre-process documents via pipelines (Grok, enrich). Config: node.roles: [ingest]. Production: separate for complex transformations.
Critical : Production must separate roles; mixing Master+Data risks GC-induced master instability and split-brain.
Shards & Replicas
A shard is a Lucene index. Primary shards (fixed at creation) are the write entry point; replicas provide HA and read scaling. Routing formula: shard_num = hash(_routing) % num_primary_shards. Changing primary count breaks routing, requiring reindex.
Shard size: 10–50 GB (author suggests 20–40 GB).
Replicas: 1–2 typical; more hurts write throughput and disk.
Primary and replica never on same node.
Write goes to primary then syncs to replicas; wait_for_active_shards controls durability.
Reads round-robin across primary/replicas for linear read scaling.
Example 3-node cluster with 3 primaries + 1 replica each:
Node 1: P0, R1
Node 2: P1, R2
Node 3: P2, R0Kill Node 1 → R0 promotes to primary instantly.
Write Path: Refresh, Translog, Flush
Write flow:
memory buffer + translog → refresh → in-memory segment (searchable) → flush → segment on disk + clear translog.
refresh : Trigger: periodic/manual. Purpose: make data searchable (NRT). Durability: no (OS cache only). Performance: medium (CPU for inversion). Default: 1s.
translog : Trigger: every write/periodic fsync. Purpose: prevent data loss on crash. Durability: yes (fsync per request by default). Performance: low (sequential write). Default: 5s or request sync.
flush : Trigger: translog full/30min/manual. Purpose: full persistence to disk. Durability: yes (segment + translog cleared). Performance: high (heavy I/O). Default: 30min or 512MB.
Common misconceptions : refresh ≠ disk persistence; translog written before flush, not after; flush ≠ refresh.
Optimization Tips
Non-real-time (logs): increase refresh_interval to 30–60s for 2–5× write throughput.
High write, lower durability: index.translog.durability: async with sync_interval: 30s.
Avoid manual flush; tune index.translog.flush_threshold_size.
Query Execution: Query Then Fetch
Default search type splits work into two phases:
Query (scatter) : Coordinator broadcasts to all relevant shards. Each shard runs the query locally, builds a priority queue, and returns only doc_id + sort fields (e.g., _score) for from+size docs. Coordinator merges and globally sorts, then truncates to final top N.
Fetch (gather) : Coordinator issues Multi-Get requests for the selected doc IDs to retrieve full _source, highlights, stored fields.
Deep pagination root cause : For from=9990, size=10, each shard returns 10,000 docs; with 10 shards, coordinator sorts 100,000 docs → memory/CPU explosion. Hence index.max_result_window defaults to 10,000.
Pagination Solutions
from+size : Max results 10,000. Jump pages: yes. Real-time: yes. Performance: fast shallow, terrible deep. Memory: high deep. Use case: web pagination (first ~100 pages).
scroll : Max results unlimited. Jump pages: no. Real-time: no (snapshot). Performance: fast full scan. Memory: high (context). Use case: bulk export, offline analysis.
search_after : Max results unlimited. Jump pages: no. Real-time: yes. Performance: stable. Memory: very low. Use case: infinite scroll, mobile feeds. search_after uses the last hit's sort values (must include a unique tie-breaker like _id) as the next query's cursor. Each shard only fetches size docs matching sort_field > last_value, so coordinator merge size stays constant ( shards * size).
GET /order/_search
{
"size": 10,
"sort": [ { "order_id": "asc" } ],
"search_after": [ 1024 ]
}Analyzers & IK Chinese Tokenizer
Analyzer pipeline: Character Filters (HTML strip, emoji replace) → Tokenizer (split into tokens) → Token Filters (lowercase, stopwords, synonyms, stemming). Index-time and search-time analyzers must match or searches miss.
IK provides two modes: ik_smart: coarse-grained (e.g., ["我", "爱", "北京", "天安门"]) ik_max_word: fine-grained exhaustive (e.g., ["我", "爱", "北京", "天安门", "天安", "门"])
Typical mapping: index with ik_max_word, search with ik_smart for precision/recall balance.
Hot Reload via Remote Dictionary
Configure IKAnalyzer.cfg.xml with HTTP endpoints:
<entry key="remote_ext_dict">http://host:port/ik/words</entry>
<entry key="remote_ext_stopwords">http://host:port/ik/stopwords</entry>IK polls every 60s; if Last-Modified changes, it reloads without restart. Backend can serve from MySQL, enabling ops to add terms via admin UI.
Common Pitfalls
Mismatched index/search analyzers → missing hits.
Missing custom dictionary → brand names split incorrectly ("小米手机" → "小","米","手机").
Wrong stopwords → important terms dropped.
Remote dict missing Content-Type: text/plain or Last-Modified header.
IK version ≠ ES version → plugin fails to load.
Debug with _analyze API and _termvectors to inspect actual indexed tokens.
Index Optimization Checklist
Design Phase (80% impact)
Shard count: target 20–40 GB/shard; shards = nodes × 1.5–3. Example: 3 nodes, 100 GB → 6 shards.
Replicas: 0 during bulk load, then 1–2 for HA.
Mappings: disable _all (ES7+ default); use keyword for exact match, index: false for non-searchable; text+keyword multi-fields for dual needs.
ILM: time-based indices (daily logs) with hot/warm/cold/delete phases.
Dynamic mapping: dynamic: strict to prevent field explosion.
Numeric types: smallest sufficient (byte/short/integer); scaled_float for prices.
Large fields: enabled: false on _source or exclude from indexing.
Classic bug: storing enum status as text then querying with term fails because text is analyzed. Fix: use keyword.
Write Optimization
Bulk API: 5–15 MB batches, ≤1000 docs.
Increase refresh_interval to 30–60s for write-heavy loads.
Merge policy: LogByteSizeMergePolicy with max_merge_at_once=10, segments_per_tier=10.
Translog: durability: async, sync_interval: 30s for higher throughput.
Query Optimization
Avoid match_all on large indices; always filter.
Routing: co-locate related docs (e.g., by merchant_id) to query single shard.
Pagination: from+size ≤10k; search_after for deep; scroll for export.
Prefer filter over query clauses: no scoring, cached, 3–10× faster.
Operations
Role separation: dedicated Master, Data, Coordinating, Ingest.
Heap: 50% RAM, max 31 GB (compressed oops limit).
SSD mandatory; disk watermark at 85%.
Monitor: shard health, merge rate, GC, disk, latency.
Scenario Patterns
Log ingestion : Throughput over real-time. Key settings: refresh_interval=30s, async translog, 10 MB bulk.
E-commerce search : Low latency, routing. Key settings: routing by user/product, filter cache, no script scoring.
Disk pressure : Compression, cleanup. Key settings: codec: best_compression, force merge read-only, rollover delete.
Frequent GC : Heap pressure. Key settings: shard <50 GB, heap <30 GB, segment count <500.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
