Elasticsearch Cluster: Inverted Index, Mechanics, Architecture for 100M+ Queries

The article starts from a realistic e‑commerce search scenario where more than 1 billion SKUs and daily query volume exceed 100 million, requiring fast, stable responses and graceful degradation.

It explains why a traditional relational database cannot meet these needs and why Elasticsearch is used as a read‑optimized projection layer, keeping the transactional source (MySQL) as the truth source.

Core Mechanisms

Inverted Index

Unlike B‑Tree indexes, an inverted index maps terms to posting lists. For example, the product title "Apple iPhone 15 Pro Max 256G" is tokenized into terms iphone, 15, pro, each pointing to document IDs.

iphone -> [doc_1001, doc_1049, doc_2033, ...]
15    -> [doc_1001, doc_2033, ...]
pro   -> [doc_1001, doc_1208, ...]

Key indexing decisions include choosing the right analyzer, deciding which fields are text vs keyword, whether to store term frequencies, positions, and using doc_values for sorting and aggregations.

Lucene Segment and Near‑Real‑Time

When a document is indexed, it passes through several stages:

Request arrives at the primary shard.

Document is analyzed and written to an in‑memory buffer.

Operation is appended to the translog for durability.

On the next refresh_interval, the buffer is flushed to a new segment, making the data searchable.

Later, a background merge combines small segments into larger ones.

This explains why a successful write does not guarantee immediate visibility and why overly aggressive refresh_interval can cause segment explosion.

Shard Routing, Replicas and Hotspots

Elasticsearch distributes an index into primary shards and replicas. Routing keys (e.g., merchant_id) determine which shard stores a document. Mis‑chosen routing can create hotspot shards, leading to CPU, heap, and I/O pressure.

Replica count improves read capacity and fault tolerance but adds write cost; it should be sized based on write‑heavy workloads, not merely for safety.

Index Modeling Guidelines

Typical product document fields: title as text with a title.raw sub‑field of type keyword for exact filtering. price as scaled_float (scaling_factor 100) for monetary precision. attrs as nested to avoid array‑object mis‑matches.

Separate keyword fields for brand, category, status, etc., to support fast filters and aggregations.

Dynamic mapping should be set to strict to prevent uncontrolled field growth.

Write Pipeline (CDC → Kafka → Projection Service → Bulk API)

In production, the recommended flow is:

Business service writes to MySQL.

Change data capture (Debezium, Canal) reads binlog.

Events are published to Kafka.

Index projector consumes events, assembles GoodsSearchDocument, and writes to Elasticsearch via the Bulk API with upsert semantics.

@Slf4j
@Service
public class GoodsIndexProjector {
    private final ElasticsearchClient client;
    private final GoodsDocumentAssembler assembler;
    private final DeadLetterPublisher deadLetterPublisher;
    // ... constructor omitted ...
    public void projectBatch(List<GoodsChangedEvent> events) {
        if (events == null || events.isEmpty()) return;
        Map<String, GoodsChangedEvent> dedup = events.stream()
            .collect(Collectors.toMap(GoodsChangedEvent::getSkuId,
                                      e -> e,
                                      this::pickLatestEvent,
                                      LinkedHashMap::new));
        BulkRequest.Builder bulkBuilder = new BulkRequest.Builder();
        List<GoodsChangedEvent> ordered = new ArrayList<>(dedup.values());
        for (GoodsChangedEvent ev : ordered) {
            GoodsSearchDocument doc = assembler.assemble(ev);
            bulkBuilder.operations(op -> op
                .update(update -> update
                    .index("goods-search-write")
                    .id(doc.getSkuId())
                    .routing(routeByMerchant(doc.getMerchantId()))
                    .action(a -> a.doc(doc).docAsUpsert(true))
                    .ifPrimaryTerm(ev.getPrimaryTerm())
                    .ifSeqNo(ev.getSeqNo())));
        }
        try {
            BulkResponse resp = client.bulk(bulkBuilder.build());
            handleBulkResponse(ordered, resp);
        } catch (Exception ex) {
            log.error("bulk project failed, size={}", ordered.size(), ex);
            ordered.forEach(deadLetterPublisher::publish);
        }
    }
    // helper methods omitted for brevity
}

The code demonstrates deduplication, idempotent upserts, version‑based conflict avoidance, and dead‑letter handling for partial failures.

Query Execution Flow

A typical search request goes through:

Coordinator node receives the request.

Request is broadcast to relevant shards.

Each shard performs inverted lookup, filters, sorting, and aggregations.

Shards return top‑N doc IDs and scores.

Coordinator merges results globally.

Coordinator fetches full documents for the final hits.

Costly steps include per‑shard topN competition, deep pagination ( from+size), and aggregations that increase memory usage.

Search Service Design

Instead of exposing raw DSL to the front‑end, the service validates input, rewrites queries, applies filters, and enforces limits. Example implementation:

@Slf4j
@Service
public class GoodsSearchService {
    private static final int MAX_PAGE_SIZE = 100;
    private final ElasticsearchClient client;
    public SearchPage<GoodsSearchDocument> search(GoodsSearchRequest request) throws IOException {
        validate(request);
        SearchResponse<GoodsSearchDocument> resp = client.search(s -> s
            .index("goods-search-read")
            .trackTotalHits(t -> t.enabled(false))
            .timeout(t -> t.time("80ms"))
            .size(request.getPageSize())
            .query(q -> q.bool(b -> {
                b.must(m -> m.multiMatch(mm -> mm
                    .query(request.getKeyword())
                    .fields("title", "brandName")));
                b.filter(f -> f.term(t -> t.field("status").value("ONLINE")));
                if (request.getCategoryId() != null) {
                    b.filter(f -> f.term(t -> t.field("categoryPath").value(request.getCategoryId())));
                }
                if (request.getMinPrice() != null || request.getMaxPrice() != null) {
                    b.filter(f -> f.range(r -> r
                        .number(n -> {
                            n.field("price");
                            if (request.getMinPrice() != null) n.gte(request.getMinPrice().doubleValue());
                            if (request.getMaxPrice() != null) n.lte(request.getMaxPrice().doubleValue());
                            return n;
                        } )));
                }
                return b;
            })
            .sort(srt -> srt.score(sc -> sc.order(SortOrder.Desc)))
            .sort(srt -> srt.field(f -> f.field("salesVolume").order(SortOrder.Desc)))
            .sort(srt -> srt.field(f -> f.field("skuId").order(SortOrder.Asc)))
            .searchAfter(request.getSearchAfter()),
            GoodsSearchDocument.class);
        List<GoodsSearchDocument> docs = resp.hits().hits().stream()
            .map(Hit::source)
            .filter(Objects::nonNull)
            .toList();
        List<FieldValue> nextAfter = resp.hits().hits().isEmpty()
            ? List.of()
            : resp.hits().hits().get(resp.hits().hits().size() - 1).sort();
        return new SearchPage<>(docs, nextAfter);
    }
    private void validate(GoodsSearchRequest req) {
        if (req == null || req.getKeyword() == null || req.getKeyword().isBlank()) {
            throw new IllegalArgumentException("keyword must not be blank");
        }
        if (req.getPageSize() <= 0 || req.getPageSize() > MAX_PAGE_SIZE) {
            throw new IllegalArgumentException("pageSize out of range");
        }
    }
}

The service enforces a maximum page size, validates required parameters, uses search_after to avoid deep pagination, and separates filtering from scoring.

Index Lifecycle Management (ILM)

ILM automates rollover, warm‑phase migration, and eventual deletion. Example policy:

PUT _ilm/policy/goods_search_ilm
{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_primary_shard_size": "40GB", "max_age": "7d" }, "set_priority": { "priority": 100 } } },
      "warm": { "min_age": "7d", "actions": { "allocate": { "include": { "data_tier": "data_warm" } }, "forcemerge": { "max_num_segments": 1 }, "set_priority": { "priority": 50 } } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Thresholds should be tuned to the actual data growth, query latency targets, and storage capacity.

Alias‑Based Index Switching

When a mapping change is required, create a new index, back‑fill data, then atomically switch the read/write aliases:

POST _aliases
{
  "actions": [
    { "remove": { "index": "goods-search-v20260701-000001", "alias": "goods-search-read" } },
    { "add":    { "index": "goods-search-v20260717-000001", "alias": "goods-search-read" } },
    { "remove": { "index": "goods-search-v20260701-000001", "alias": "goods-search-write" } },
    { "add":    { "index": "goods-search-v20260717-000001", "alias": "goods-search-write", "is_write_index": true } }
  ]
}

This ensures zero‑downtime migration and easy rollback.

Common Pitfalls and Optimizations

Deep pagination : from+size forces each shard to collect large candidate sets; use search_after instead.

Hot shards : improper routing creates uneven shard sizes; consider custom routing or splitting hot data into separate index families.

Fielddata & aggregations on text fields : cause high heap usage; prefer keyword or doc_values for sorting/aggregations.

Merge storms : large bulk writes can saturate I/O; monitor refresh_interval, merge backlog, and tune bulk parameters together with segment size.

Cluster recovery : after node restarts, limit concurrent recoveries and throttle I/O to avoid service degradation.

Monitoring, Alerting and Capacity Planning

Four monitoring dimensions are recommended:

Query side: QPS, latency percentiles, slow‑query sampling, timeout counts, high‑cost DSL distribution.

Write side: bulk success/failure rates, reject counts, write latency, CDC lag, dead‑letter volume.

Node/Shard side: CPU, heap, GC time, disk I/O, segment count, shard balance, recovery progress.

Lifecycle/control plane: cluster health, unassigned shards, master switches, ILM phase stalls, cluster‑state size.

Alerting should be chain‑aware (e.g., CDC lag increase, bulk failures, query tail latency) to pinpoint the failing segment of the pipeline.

Evolution Roadmap

The article outlines a staged evolution:

Stage 1 : Single index, basic keyword search, direct CDC → ES.

Stage 2 : Introduce aliases, index templates, gray‑scale reindexing.

Stage 3 : Add hot/warm tiers, ILM, query protection (limits, rate‑limiting, fallback).

Stage 4 : Platformize with configurable query templates, versioned analyzers, A/B testing, automated rebuild & rollback.

Production Checklist

Confirm Elasticsearch is a read‑only projection; the source of truth remains the transactional DB.

Ensure a reliable CDC → Kafka → projection service pipeline with dead‑letter handling.

Define aliases, templates, and ILM policies for core indices; have rollback plans.

Guard against deep pagination, wildcard queries, large aggregations, and expensive scripts.

Separate fields for search, filter, and sort; avoid using text for aggregations.

Implement search_after, async export, and query white‑listing in the service layer.

Monitor the four metric groups listed above; set alerts that map to specific pipeline stages.

Practice index rebuilds, alias switches, dead‑letter replay, node recovery, and snapshot restores.

Communicate the eventual‑consistency visibility window to business owners.

By treating Elasticsearch as a dedicated search projection layer and applying the disciplined engineering practices described, teams can reliably support hundred‑million‑query‑per‑day workloads with predictable latency and robust operational safety.