Production‑grade Elasticsearch: From Lucene Internals to Billion‑scale Search Architecture

Why Elasticsearch Becomes Fragile in Production

In development environments Elasticsearch feels "friendly": creating an index is fast, ingesting tens of thousands of documents is quick, match returns results instantly, and Kibana shows everything as normal. However, once traffic scales, problems explode:

Indexes grow rapidly, causing shard count to spiral out of control.

Refresh spikes during write peaks generate many small segments, increasing merge pressure.

Mixed full‑text, filter, sort and aggregation queries drive coordinating node CPU to the roof.

Continuous addition of business fields leads to "field explosion".

Heap may appear sufficient, but Page Cache becomes a bottleneck and search latency jitters.

When a node drops, replica reallocation triggers a second I/O storm.

The common root cause is treating Elasticsearch as a "full‑text enabled database" instead of a distributed search system built on Lucene. Without a proper architectural view, teams inevitably hit shard governance, resource isolation, index lifecycle, and fault‑tolerance challenges.

Business System
  -> How data changes enter ES
  -> How indexes are organized and rolled
  -> How queries are executed and isolated
  -> How the cluster carries hot and cold data
  -> How failures are limited to local scope

1. Lucene Fundamentals that Drive Online Performance

1.1 Inverted Index is a "term‑to‑document" system, not a "document‑to‑field" index

Relational DB indexes map field values to records. Lucene’s core maps terms to the list of documents that contain them.

doc1: "red running shoes"
doc2: "running socks for sport"

Inverted index looks like:

"red"   -> [doc1]
"running" -> [doc1, doc2]
"shoes" -> [doc1]
"socks" -> [doc2]
"sport" -> [doc2]

Lucene maintains several layers:

term dictionary – efficient term lookup

postings list – list of docs per term

term frequency – relevance scoring

positions/offsets – phrase queries, highlighting

doc values – column‑store for sorting, aggregations, scripts

stored fields / _source – original document source

This determines that mapping design is a trade‑off between query capability and resource cost.

1.2 refresh, flush, merge – why a successful write is not instantly searchable

Elasticsearch is near‑real‑time, not strong‑consistent. The write path can be simplified as:

Client write
  -> Coordinating node routes to primary shard
  -> Primary shard writes to in‑memory buffer + translog
  -> Periodic refresh creates searchable segment
  -> Periodic or threshold‑based flush persists segment
  -> Background merge combines small segments

Key actions: refresh: makes new docs visible but does not fully persist. flush: persists translog to disk. merge: combines small segments to reduce query fan‑out and delete cost.

Production facts:

Higher write throughput creates segments faster.

More fragments increase the number of segments a query must scan.

Frequent merges saturate disk I/O and CPU.

Frequent refreshes hurt bulk write efficiency.

Typical bulk‑import strategy:

Temporarily increase refresh_interval.

Reduce or disable replicas.

Write in bulk batches.

Monitor merge backlog and disk activity.

Restore normal refresh and replica settings after import.

1.3 Shards are the fundamental capacity unit, not just a logical concept

Each query that hits 12 primary shards and 12 replicas actually runs 24 concurrent Lucene instances.

Shard count directly influences:

How many execution units a single query fans out to.

How much result aggregation the coordinating node must perform.

Data migration volume during cluster recovery.

Network and disk bandwidth needed for primary‑replica rebuild.

Too many shards cause memory, file‑handle, cache, and thread‑scheduling overhead, as well as excessive segment count and merge inefficiency.

1.4 Relevance, filter, sort, and aggregation follow different execution paths

Many slow queries are not due to complex DSL but because high‑cost operations are mixed together.

Four capability categories:

Full‑text search – relies on inverted index and relevance scoring.

Filter – should use filter context to avoid scoring.

Sort – prefer doc_values fields, avoid script sorting.

Aggregation – bucket‑based statistics, often more memory‑intensive than the search itself.

A production‑friendly query structure:

must: full‑text conditions
filter: non‑scoring conditions
sort: explicit sort fields
size / search_after: pagination limit
aggs: only when needed

Putting aggregation, deep pagination, highlighting, and large fields together will overload any hardware.

2. Production Architecture Perspective – Build a Complete Search System

A robust search system consists of four layers:

Access Layer
  -> How business apps write, query, auth, rate‑limit
Index Layer
  -> Index templates, mapping, aliases, rollover, lifecycle
Execution Layer
  -> Node roles, shard layout, hot‑spot isolation, query routing, bulk write
Governance Layer
  -> Monitoring, alerting, capacity planning, fault recovery, data validation, rollback

2.1 Real‑world e‑commerce search architecture

Assume:

80 million products

3 million daily increments

QPS > 10 000 during promotions

Frequent price, stock, and status updates

Support keyword search, filter, sort, aggregation, suggestions, operational interventions

Typical layered deployment:

MySQL / Product Service
  -> Binlog / Business events
  -> Kafka
  -> Index Builder Service
  -> Elasticsearch write alias

User request
  -> API Gateway
  -> Search Service
  -> Elasticsearch read alias
  -> Result re‑ranking / operational rules / cache

Two often‑overlooked components: Index Builder Service: not just a Kafka consumer – it handles model conversion, field trimming, idempotency, batch processing, retry, and dead‑letter handling. Data Repair / Rebuild: when messages are lost, mapping changes, analyzer updates, or sort logic rewrites occur, a dedicated repair pipeline is required instead of relying on the live consumer to self‑heal.

2.2 Node‑role separation is about isolation, not elegance

Production clusters should not let every node perform every role. Recommended roles: master: manages cluster state only, no data. data_hot: handles high‑frequency reads/writes and hot indexes. data_warm: stores low‑frequency historical data. coordinating only: receives queries and aggregates results. ingest: runs ingest pipelines or pre‑processing.

Benefits:

Large aggregations no longer compete for master resources.

Write spikes stay away from historical query nodes.

Hot and cold data can be placed on different hardware tiers.

Scaling can target a single role instead of whole‑cluster scaling.

2.3 Hot‑spot problems cannot be solved by merely adding machines

Hot spots arise from:

Popular keywords causing query hot spots.

High‑frequency product updates causing write hot spots.

Time‑based skew where the newest index receives the majority of traffic.

Horizontal scaling alone only increases average resources, not local hot‑spot pressure. Effective mitigations:

Use routing to spread writes across primary shards.

Separate read/write aliases to decouple query load from indexing.

Cache popular query results.

Split large aggregations from normal search APIs.

Model high‑frequency update fields separately from long‑tail display fields.

3. Index Modeling – Mapping Design Determines System Limits

3.1 Define field purpose before choosing type

Ask four questions for each field before it enters ES:

Is it a full‑text searchable field or an exact‑filter field?

Does it participate in sorting or aggregation?

Does it need highlighting or phrase matching?

Must it be retained in _source?

Typical field strategies:

IDs, category IDs, brand IDs → keyword (exact filter, aggregation, sort friendly)

Product title → text + multi‑fields for keyword and pinyin

Status, small enums → byte / short (space‑efficient)

Price, sales count, stock → numeric types (cheaper for range queries and sorting)

Tag arrays → keyword (filter & aggregation friendly)

Long description → text only, avoid sorting/aggregation

3.2 Multi‑field design is the norm in production

One title field often needs:

Full‑text search

Exact deduplication

Pinyin search

Operational weighted sorting

Typical mapping snippet:

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "productId": {"type": "keyword"},
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart",
        "fields": {
          "keyword": {"type": "keyword", "ignore_above": 256},
          "pinyin": {"type": "text", "analyzer": "pinyin_index", "search_analyzer": "pinyin_search"}
        }
      },
      "brandId": {"type": "keyword"},
      "categoryId": {"type": "keyword"},
      "price": {"type": "long"},
      "status": {"type": "byte"},
      "salesCount": {"type": "long"},
      "updatedAt": {"type": "date"}
    }
  }
}

3.3 Strongly enforce strict dynamic mapping

Many index crashes stem from uncontrolled field growth:

Business adds ad‑hoc attributes directly into ES.

Different teams add divergent fields without naming standards.

A/B experiment fields linger after the test.

Consequences:

Cluster state balloons, slowing metadata propagation.

Field structures consume ever‑growing memory and disk.

Production recommendations:

Disable permissive dynamic mapping unless absolutely necessary.

Validate fields against a whitelist at the ingestion point.

Consolidate extensible attributes into a structured sub‑object rather than flat, unlimited fields.

Example of strict mapping:

{
  "mappings": {
    "dynamic": "strict"
  }
}

3.4 Object vs. nested vs. denormalization – trade‑offs

Complex business models (e.g., products with multiple specifications) require one of three approaches: object: simple structure but arrays are flattened, risking incorrect matches. nested: semantically correct but higher query cost.

Denormalization: expand frequently searched dimensions directly into the product document; fastest reads but more complex writes.

In high‑throughput e‑commerce search, denormalization is usually preferred because reads dominate writes, latency matters more than modeling elegance, and ES is not suited for complex relational joins.

4. Write‑Path Design – Stable, Recoverable Ingestion

4.1 Prefer CDC or business events over synchronous double‑write

Many teams start with: Transaction commit -> synchronous ES write call This leads to:

Database success but ES failure → data inconsistency.

ES latency spikes slow down the main transaction path.

Retry logic pollutes business services.

A more robust approach:

Business DB write success
  -> Binlog / Outbox / Domain event
  -> Kafka
  -> Index Builder Service
  -> Bulk write to ES

Advantages:

Decouples primary business flow from search indexing.

Enables bulk writes for higher throughput.

Supports independent retry, compensation, and replay.

Allows pre‑write field trimming, model conversion, and operational rule injection.

4.2 Idempotency is the baseline for ES writes

With at‑least‑once delivery, the indexer must be idempotent:

Use stable business IDs as document IDs instead of random UUIDs.

Include version or update timestamp inside the document.

Only overwrite when the incoming event version is newer than the stored version.

Deletion events must be explicit, not inferred from missing docs.

Example document:

{
  "productId": "P202607050001",
  "title": "Lightweight breathable running shoes",
  "status": 1,
  "version": 183,
  "updatedAt": "2026-07-05T10:00:00Z"
}

4.3 Bulk write is a trade‑off between throughput and recoverability

Bulk writes dramatically increase throughput but size matters:

Too large a request overloads the coordinating node.

Large batches increase failure cost.

Retrying a huge batch spikes disk and thread‑pool load.

Production‑grade control dimensions:

Number of documents per batch.

Byte size per batch.

Concurrent batch count.

Average latency and failure rate.

Recommended settings:

Batch size: a few hundred to a few thousand documents.

Payload: 5 MB – 15 MB.

Scale concurrency based on load tests and thread‑pool rejection rates, not by intuition.

4.4 Production‑grade Java write example (Spring Boot 3)

The following code demonstrates connection pooling, timeouts, bulk handling, retry, partial‑failure collection, and dead‑letter persistence.

package com.example.search.config;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.UsernamePasswordCredentials;
import org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient;
import org.apache.hc.client5.http.impl.async.HttpAsyncClients;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.util.Timeout;
import org.elasticsearch.client.RestClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.List;

@Configuration
@Bean(destroyMethod = "close")
public class ElasticsearchClientConfig {
    public RestClient restClient(@Value("${search.es.hosts}") List<String> hosts,
                               @Value("${search.es.username}") String username,
                               @Value("${search.es.password}") String password) {
        BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(new AuthScope(null, -1),
                new UsernamePasswordCredentials(username, password.toCharArray()));
        HttpHost[] httpHosts = hosts.stream().map(HttpHost::create).toArray(HttpHost[]::new);
        return RestClient.builder(httpHosts)
                .setRequestConfigCallback(config -> config
                        .setConnectTimeout(Timeout.ofSeconds(3))
                        .setResponseTimeout(Timeout.ofSeconds(10)))
                .setHttpClientConfigCallback(client -> {
                    CloseableHttpAsyncClient asyncClient = HttpAsyncClients.custom()
                            .setDefaultCredentialsProvider(credentialsProvider)
                            .setMaxConnTotal(200)
                            .setMaxConnPerRoute(100)
                            .build();
                    asyncClient.start();
                    return asyncClient;
                })
                .build();
    }

    @Bean
    public ElasticsearchClient elasticsearchClient(RestClient restClient) {
        return new ElasticsearchClient(new RestClientTransport(restClient, new JacksonJsonpMapper()));
    }
}

package com.example.search.index;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.Refresh;
import co.elastic.clients.elasticsearch.core.BulkRequest;
import co.elastic.clients.elasticsearch.core.BulkResponse;
import co.elastic.clients.elasticsearch.core.bulk.BulkResponseItem;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

@Slf4j
@Service
@RequiredArgsConstructor
public class ProductIndexWriter {
    private static final int MAX_BATCH_SIZE = 1000;
    private final ElasticsearchClient client;
    private final FailedDocumentStore failedDocumentStore;

    public void bulkUpsert(List<ProductIndexDocument> documents) {
        if (documents == null || documents.isEmpty()) {
            return;
        }
        List<List<ProductIndexDocument>> partitions = partition(documents, MAX_BATCH_SIZE);
        for (List<ProductIndexDocument> batch : partitions) {
            executeWithRetry(batch, 3);
        }
    }

    private void executeWithRetry(List<ProductIndexDocument> batch, int maxRetry) {
        int attempt = 0;
        while (attempt < maxRetry) {
            attempt++;
            try {
                BulkResponse response = doBulk(batch);
                if (!response.errors()) {
                    return;
                }
                List<ProductIndexDocument> failedDocs = collectFailedDocuments(batch, response);
                log.warn("bulk write partially failed, attempt={}, failed={}", attempt, failedDocs.size());
                batch = failedDocs;
            } catch (Exception ex) {
                log.error("bulk write failed, attempt={}", attempt, ex);
            }
            sleepBackoff(attempt);
        }
        failedDocumentStore.persist(batch);
        log.error("bulk write permanently failed, batch persisted to local failed store, size={}", batch.size());
    }

    private BulkResponse doBulk(List<ProductIndexDocument> batch) throws IOException {
        BulkRequest.Builder builder = new BulkRequest.Builder();
        for (ProductIndexDocument doc : batch) {
            builder.operations(op -> op.index(idx -> idx
                    .index(resolveWriteIndex())
                    .id(doc.getProductId())
                    .document(doc)));
        }
        return client.bulk(builder.refresh(Refresh.False).build());
    }

    private List<ProductIndexDocument> collectFailedDocuments(List<ProductIndexDocument> originalBatch, BulkResponse response) {
        List<ProductIndexDocument> failed = new ArrayList<>();
        List<BulkResponseItem> items = response.items();
        for (int i = 0; i < items.size(); i++) {
            BulkResponseItem item = items.get(i);
            if (item.error() != null) {
                failed.add(originalBatch.get(i));
                log.warn("document write failed, id={}, reason={}", originalBatch.get(i).getProductId(), item.error().reason());
            }
        }
        return failed;
    }

    private String resolveWriteIndex() {
        return "products-write";
    }

    private void sleepBackoff(int attempt) {
        try {
            Thread.sleep(Math.min(1000L * attempt, 3000L));
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }

    private List<List<ProductIndexDocument>> partition(List<ProductIndexDocument> source, int size) {
        List<List<ProductIndexDocument>> result = new ArrayList<>();
        for (int i = 0; i < source.size(); i += size) {
            result.add(source.subList(i, Math.min(i + size, source.size())));
        }
        return result;
    }
}

Key takeaways from the code:

Stable business IDs provide natural upsert idempotency.

Batch retries are limited; partial failures are isolated and persisted for later replay.

Failed documents are stored in a dead‑letter store for manual or automated compensation.

4.5 Index rebuild must be a standard capability, not an emergency fix

When any of the following occurs, rebuild rather than patch the live index:

Analyzer changes.

New fields added to the retrieval set.

Sorting strategy overhaul.

Incompatible mapping upgrades.

Document structure slimming.

Standard rebuild workflow:

Create new index
  -> Replay full data
  -> Dual‑write or incremental catch‑up
  -> Validate document count and sample queries
  -> Switch read alias
  -> Then switch write alias
  -> Retire old index after a grace period

Alias switching is a core governance capability; without it, rebuilding becomes a high‑risk operation.

5. Query‑Path Design – Balancing Quality, Latency, and Resource Use

5.1 Split search requests into four semantic layers

A mature search service does not forward raw front‑end parameters directly to DSL. Instead it separates:

Query intent – keywords, categories, brands, price range, sort order.

Query rewrite – synonym expansion, stop‑word removal, operational black/white lists.

Execution – ES DSL, pagination, aggregations, timeout control.

Result governance – caching, result shuffling, operational boosts, graceful degradation.

This keeps ES focused on what it does best (retrieval and basic sorting) while business logic lives in the service layer.

5.2 Production‑ready DSL example for product search

{
  "query": {
    "bool": {
      "must": [
        {"multi_match": {"query": "lightweight running shoes", "fields": ["title^4", "title.pinyin^2", "description"]}}
      ],
      "filter": [
        {"term": {"status": 1}},
        {"term": {"categoryId": "shoes"}},
        {"range": {"price": {"gte": 30000, "lte": 80000}}}
      ]
    }
  },
  "sort": [
    {"_score": "desc"},
    {"salesCount": "desc"},
    {"productId": "asc"}
  ],
  "size": 20
}

Key points:

Full‑text conditions belong in must.

Exact filters belong in filter.

Sort on doc_values ‑friendly fields.

Always add a stable tie‑breaker (e.g., productId) to avoid pagination jitter.

5.3 Deep pagination must use search_after , not from+size

When page numbers become large, from+size forces each shard to build a huge candidate set before the coordinating node can sort, leading to massive overhead. Preferred solutions:

Use search_after with the sort values of the last hit.

Provide a separate offline export API for full data dumps.

Limit user‑visible pagination depth.

5.4 Query cache solves repeated requests but not fundamentally bad queries

Cache is an accelerator, not a safety net. Suitable scenarios:

Hot keyword searches on the homepage.

Fixed filter combinations on operational channel pages.

Low‑frequency aggregations on brand or category.

Unsuitable scenarios:

Long‑tail complex queries.

Highly personalized queries with unique parameters each time.

Real‑time sorting that depends on rapidly changing fields.

If the underlying DSL is poorly designed (excessive fields, heavy aggregations, unsuitable sorting, or unseparated filter), caching only masks the symptom.

5.5 Search service must support graceful degradation

When ES experiences:

Partial shard timeouts.

Aggregation timeouts.

High‑cost highlighting.

Suggestion service failures.

Recommended fallback strategies:

Return basic results, disable highlighting and heavy aggregations.

Switch popular category results to cached data.

After timeout, keep only the core sort dimension.

Lazily load low‑priority filter items.

6. Index Lifecycle & Data Tiering – Keep the Cluster Controllable

6.1 Not all data belongs on hot nodes

Across product search, log search, and content search, the pattern is:

Recent data is accessed most frequently.

Historical data access probability decays over time.

Keeping everything on high‑performance nodes causes linear cost growth.

Recommended tiering:

Hot – recent, high‑frequency write & search data, stored on SSD.

Warm – older data with reduced write rate, stored on regular high‑performance cloud disks.

Cold – low‑frequency historical data, stored on low‑cost storage.

6.2 Use aliases and rolling indices instead of a single massive index

Writing to a single ever‑growing index leads to:

Difficult segment management.

Inflexible shard scaling.

Expensive data migration and rebuild.

Coarse‑grained rollback.

Best practice:

Write only via a write alias.

Background jobs roll new indices by time or size.

Read through a read alias that points to the current hot/warm indices.

Example index template and initial index with aliases:

PUT _index_template/products_template
{
  "index_patterns": ["products-*"],
  "template": {
    "settings": {
      "number_of_shards": 6,
      "number_of_replicas": 1
    }
  }
}

PUT products-000001
{
  "aliases": {
    "products-write": {"is_write_index": true},
    "products-read": {}
  }
}

6.3 ILM is about standardizing governance, not flashy automation

In production, ILM turns manual ops into policy‑driven actions. A stable approach:

Hot phase – rollover when primary shard reaches 40 GB or 1 day.

Warm phase (after 7 days) – force‑merge to a single segment.

Delete phase (after 90 days) – automatic deletion.

Sample ILM policy:

PUT _ilm/policy/products_ilm
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {"max_primary_shard_size": "40gb", "max_age": "1d"}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {"forcemerge": {"max_num_segments": 1}}
      },
      "delete": {
        "min_age": "90d",
        "actions": {"delete": {}}
      }
    }
  }
}

Policy tuning must match business reality:

Never force‑merge a high‑write index too early.

Do not move historically hot indexes to warm too soon.

Deletion periods should align with audit requirements, not arbitrary 30‑day defaults.

7. Capacity Planning – Beyond “How Many Machines?”

7.1 Five dimensions to consider

Raw data volume.

Index inflation factor.

Replica count.

Peak write throughput.

Peak query concurrency.

7.2 Realistic estimation model for a product index

Assumptions:

Average raw JSON = 3 KB.

Index inflation = 1.5 – 3× after inverted index, doc values, _source.

1 replica.

Daily increment = 3 million docs.

Storage estimate:

Daily raw data = 3 M × 3 KB ≈ 8.6 GB
× 2 (inflation) ≈ 17.2 GB
× 2 (replica) ≈ 34.4 GB per day
× 90 days ≈ 3 TB total

Additional overheads (segment merges, snapshots, rebuilds, OS reserve, logs, monitoring) require a safety buffer beyond the raw calculation.

7.3 Shard planning matters more than node count

Goal: keep each primary shard size in a reasonable range – not too small (excessive shard count) and not too large (slow recovery, migration). Evaluate:

How many days of data a rolling index should hold.

Target primary shard size based on historical growth and load tests.

Whether fan‑out under peak traffic is acceptable.

7.4 Heap is not the only memory concern – Page Cache is critical

Many teams focus solely on JVM heap, ignoring that ES query performance heavily depends on OS file system cache. Principles:

JVM heap should not be oversized; keep it below 30‑40 % of total RAM.

Reserve sufficient RAM for Page Cache to hold hot segments.

Node memory planning must consider disk type, segment hotness, and cache size together.

If heap is huge but Page Cache is starved, GC may look fine while query latency constantly jitters.

8. Monitoring & Alerting – Detect, Diagnose, Recover

8.1 Four core metric groups

Cluster health & shard state

cluster health

unassigned shards

relocating shards

pending tasks

Query & write latency

indexing latency

search latency

bulk rejection rate

thread‑pool queue / rejected count

Resource usage

heap usage

old / young GC frequency

CPU utilization

disk usage

filesystem cache metrics

Index internal metrics

segment count

merge time

refresh time

field data / request breaker usage

cache hit / miss rates

8.2 Alerts must point to concrete remediation steps

Bad alerts only say “ES is alerting”. Good alerts include actionable guidance: unassigned_shards > 0 → check node status, disk watermarks, run allocation explain. thread_pool.search.rejected rising → investigate large aggregations or slow queries, then examine coordinating node CPU. heap > 85% with frequent old GC → identify whether fielddata, aggregations, scripts, or mapping bloat are the cause. merge time consistently high → review write batch size, refresh strategy, and disk I/O.

8.3 Slow‑query governance should be continuous

Common slow‑query root causes:

Returning too many fields.

Large highlighted snippets.

Broad aggregations.

Script‑based sorting.

Deep pagination.

Stacking multiple high‑cost operations.

Remediation usually involves:

Trim returned fields.

Move heavy aggregations to a dedicated endpoint.

Downgrade highlighting and suggestion features.

Refactor DSL and improve index modeling.

9. Common Production Incidents & Handling

9.1 Incident 1 – Cluster turns Yellow, writes start jittering

Typical causes:

Replica allocation failure.

A data node goes offline.

Disk exceeds high water mark.

Investigation steps:

Identify which shards are unassigned.

Check node health for failures.

Inspect disk watermarks and run allocation explain.

Determine if the issue is transient or a capacity shortage.

Key points:

Avoid blind reroute commands.

Assess whether a rebalance storm would be triggered.

During peak traffic, consider limiting migrations to keep the service available.

9.2 Incident 2 – Query latency spikes while CPU is not saturated

Often mis‑diagnosed as “nothing is wrong”. Real bottlenecks are usually disk I/O or Page Cache.

Typical triggers:

Merge backlog causing disk contention.

Excessive segment count.

Hot data not cached.

Expensive highlighting on large fields.

Resolution direction:

Inspect segment count and merge backlog.

Check disk I/O wait metrics.

Verify recent bulk imports or index rebuilds.

Apply quick degradation on high‑cost query paths.

9.3 Incident 3 – After index rebuild and traffic switch, sorting behaves incorrectly

“Document count matches” does not guarantee identical behavior.

Root causes:

Analyzer version or configuration change.

Missing or type‑changed fields.

Absent stable tie‑breaker causing pagination instability.

Operational weight field not fully back‑filled.

Pre‑switch validation checklist:

Validate total document count.

Run sampled queries and compare results.

Execute regression checks on core business keyword set.

10. Evolution Roadmap – From Single Node to Production‑grade System

10.1 Stage 1 – Single‑node validation

Use cases: local development, proof‑of‑concept, tiny data volumes.

Goal: verify search model, mapping correctness, and end‑to‑end product search flow – not performance tuning.

10.2 Stage 2 – Small‑scale high‑availability

Use cases: single business line, low daily active users, limited query/write scale.

Focus:

Introduce primary‑replica.

Set up basic monitoring.

Use aliases for index switching.

10.3 Stage 3 – Role separation & write‑path decoupling

Use cases: growing query and write volume.

Key actions:

Move to CDC / message‑queue based indexing.

Implement bulk writes.

Separate master, data, and coordinating nodes.

10.4 Stage 4 – Hot‑warm tiering & standardized governance

Use cases: data volume continuously grows, historical data dominates.

Focus:

Deploy ILM.

Separate hot and warm nodes.

Standardize capacity planning and slow‑query handling.

10.5 Stage 5 – Multi‑cluster & cross‑region disaster recovery

Use cases: multi‑region business, data‑center‑level fault tolerance, search becomes core infrastructure.

Key actions:

Cross‑cluster search or replication.

Traffic‑switching runbooks.

Snapshot‑restore drills.

Isolate different business domains into separate clusters.

A mature organization runs not a single massive ES cluster but a suite of search infrastructures isolated by data, traffic, and failure domains.

11. Production Checklist

11.1 Index design checklist

Explicitly differentiate text, keyword, numeric, and date fields.

Disable unnecessary dynamic field expansion.

Design sorting and aggregation fields with doc_values support.

Avoid modeling complex relational data directly in ES.

11.2 Write‑path checklist

Migrate from synchronous double‑write to asynchronous index building.

Ensure idempotency, retry, dead‑letter, and compensation mechanisms.

Support standardized index rebuild and alias switch workflow.

Have a fallback mechanism for failed data replay.

11.3 Query‑path checklist

Avoid deep pagination.

Separate filter clauses from full‑text queries.

Limit the number of returned fields.

Provide degradation paths for heavy aggregations, highlighting, and large result sets.

11.4 Cluster governance checklist

Implement role separation.

Apply hot‑warm data tiering.

Maintain capacity model and baseline shard planning.

Deploy comprehensive monitoring, alerting, slow‑query governance, and recovery drills.