3 Billion Products, 500 Million Daily Queries: How I Boosted a Failing Elasticsearch Cluster 5×

Problem Portrait – Why the Cluster Crashed

During a major promotion the 30‑node Elasticsearch cluster handling 3 billion product documents and ~5 × 10⁸ daily queries exhibited:

P99 latency rose from 80 ms to 3.2 s.

Search thread‑pool rejections (es_rejected_execution_exception) surged.

Frequent Full GC pauses caused node isolation and shard rebalance.

Price/stock updates lagged up to 45 minutes, with Kafka back‑pressure.

Root causes were identified in five layers:

Incorrect query model (deep pagination, wildcard‑only queries, heavy script_score, high‑cardinality aggregations).

Uncontrolled dynamic mapping leading to field explosion and large cluster‑state.

Write and read workloads sharing the same resource pool.

Missing application‑level traffic guards.

Inadequate index design and shard sizing.

Understanding Elasticsearch Execution

Query Phase

Each shard independently selects its Top‑N based on from + size. Larger from values increase per‑shard candidate sets, sorting cost, and coordination overhead.

Fetch Phase

After the coordinating node merges global Top‑N, it fetches _source, stored fields, or highlights from the relevant shards. Large _source or many fields increase fetch cost.

Write Path

Document enters an in‑memory buffer.

Operation is recorded in the translog.

On refresh a searchable segment is created.

Background merge threads continuously combine small segments.

High‑frequency writes therefore cause frequent refreshes, segment proliferation, and merge‑induced I/O.

Governance Layer – Query Guard Before Elasticsearch

Key insight: Never hand every request directly to Elasticsearch. A query‑governor sits between the API gateway and ES and performs:

Parameter guards (limit from+size, aggregation bucket count, returned fields).

Semantic guards (reject match_all without filters, wildcard‑only queries, dangerous scripts).

Capacity guards (auto‑circuit‑break when rejection rate rises).

Cache guards (hot terms/categories served from cache).

Observability guards (record DSL fingerprint, latency, hit‑rate, degradation reason).

Request flow:

User Request → API Gateway → Search Service → Query Governor → (cache / degrade / block) → Elasticsearch → Rule Engine → Result Merge → Response

Core Guard Code (Spring Boot example)

@Component
public class SearchGuard {
    private static final int MAX_PAGE_WINDOW = 2000;
    private static final int MAX_AGG_BUCKETS = 1000;

    public void validate(ProductSearchRequest request) {
        if (request.getPageSize() <= 0 || request.getPageSize() > 100) {
            throw new IllegalArgumentException("pageSize out of range");
        }
        if (request.getFrom() + request.getPageSize() > MAX_PAGE_WINDOW && request.getSearchAfterToken() == null) {
            throw new IllegalArgumentException("deep pagination must use search_after");
        }
        if (request.isMatchAllWithoutFilter()) {
            throw new IllegalArgumentException("match_all without filters is forbidden");
        }
        if (request.getAggBucketSize() != null && request.getAggBucketSize() > MAX_AGG_BUCKETS) {
            throw new IllegalArgumentException("agg bucket too large");
        }
        if (request.containsWildcardPrefixAndSuffix()) {
            throw new IllegalArgumentException("wildcard query is forbidden");
        }
    }
}

Degradation Logic (Sentinel fallback)

@Service
public class SearchFacade {
    private final ProductSearchService productSearchService;
    private final SearchCacheService cacheService;

    @SentinelResource(value = "product-search", blockHandler = "fallback")
    public SearchPage<ProductCard> query(ProductSearchRequest request) {
        return productSearchService.search(request);
    }

    public SearchPage<ProductCard> fallback(ProductSearchRequest request, BlockException ex) {
        SearchPage<ProductCard> cached = cacheService.getHotQueryCache(request.cacheKey());
        if (cached != null) {
            return cached.withDegraded(true);
        }
        return SearchPage.emptyDegraded("system busy, fallback triggered");
    }
}

Architecture Upgrade – From Direct ES Calls to a Search Platform Prototype

Asynchronous bulk write pipeline : MySQL binlog → Canal/Flink CDC → Kafka → Index Builder → Bulk Processor → ES.

Read‑write separation : Dedicated write master cluster for real‑time ingestion; query‑only replica cluster (CCR or async sync) for searches.

ILM hot‑warm‑cold lifecycle to keep recent hot data on SSD, warm data on cheaper storage, and archive cold data.

Bulk Processor Configuration (Production‑grade)

@Bean(destroyMethod = "close")
public BulkProcessor bulkProcessor(RestHighLevelClient client) {
    BulkProcessor.Listener listener = new BulkProcessor.Listener() {
        @Override public void beforeBulk(long id, BulkRequest request) {
            log.info("before bulk, id={}, actions={}", id, request.numberOfActions());
        }
        @Override public void afterBulk(long id, BulkRequest request, BulkResponse resp) {
            if (resp.hasFailures()) {
                log.error("bulk partially failed, id={}, msg={}", id, resp.buildFailureMessage());
            }
        }
        @Override public void afterBulk(long id, BulkRequest request, Throwable failure) {
            log.error("bulk failed, id={}", id, failure);
        }
    };
    return BulkProcessor.builder((br, bl) -> client.bulkAsync(br, RequestOptions.DEFAULT, bl), listener)
        .setBulkActions(3000)
        .setBulkSize(new ByteSizeValue(16, ByteSizeUnit.MB))
        .setFlushInterval(TimeValue.timeValueSeconds(1))
        .setConcurrentRequests(2)
        .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(200), 5))
        .build();
}

Pitfalls: setBulkActions too large amplifies failure cost. setConcurrentRequests too high saturates the write thread pool.

Always configure retry and dead‑letter queues; ES is never 100 % stable.

Index Design – Strict Schema and Controlled Attributes

Dynamic mapping was disabled and a strict template was introduced. Attributes are stored in a nested attrs array, keyword fields are used for filters/sorts, and scaled_float stores prices.

PUT _index_template/product-template
{
  "index_patterns": ["product-*"],
  "template": {
    "settings": {
      "number_of_shards": 24,
      "number_of_replicas": 1,
      "refresh_interval": "30s",
      "sort.field": ["category_id", "popularity_score"],
      "sort.order": ["asc", "desc"]
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "spu_id": {"type": "keyword"},
        "sku_id": {"type": "keyword"},
        "category_id": {"type": "keyword", "doc_values": true},
        "brand_id": {"type": "keyword", "doc_values": true},
        "title": {"type": "text", "analyzer": "ik_max_word", "search_analyzer": "ik_smart", "fields": {"raw": {"type": "keyword", "ignore_above": 256}}},
        "price": {"type": "scaled_float", "scaling_factor": 100},
        "stock": {"type": "integer"},
        "status": {"type": "byte"},
        "popularity_score": {"type": "float"},
        "update_time": {"type": "date"},
        "attrs": {"type": "nested", "properties": {"name": {"type": "keyword"}, "value": {"type": "keyword"}}}
      }
    }
  }
}

Benefits:

No field explosion; cluster‑state stays small.

Keyword fields enable efficient filtering, aggregation, and sorting. scaled_float halves storage compared to double.

Clear separation of searchable vs. display‑only fields.

Shard Sizing – Balanced Shard Size

Original index used 5 primary shards (~200 GB each). It was re‑sharded to 24‑30 primary shards, keeping each shard between 20 GB and 50 GB. This reduced merge cost, improved parallel query execution, and shortened rebalance times.

Query Optimizations

Replace deep pagination with search_after for front‑end and scroll / point‑in‑time for back‑office.

Move non‑scoring clauses (brand, category, price range, stock, self‑operated flag) into filter context to leverage filter cache.

Eliminate wildcard queries and heavy script_score; prefer match_phrase_prefix or edge_ngram.

Limit terms aggregations to lightweight front‑end buckets; offload heavy analytics to ClickHouse/Doris.

Set track_total_hits_up_to(10000) to avoid full count.

Search Service Example (Java)

@Service
public class ProductSearchService {
    private final RestHighLevelClient client;
    public SearchPage<ProductCard> search(ProductSearchRequest request) {
        SearchSourceBuilder source = new SearchSourceBuilder();
        BoolQueryBuilder bool = buildQuery(request);
        source.query(bool);
        source.size(request.getPageSize());
        source.trackTotalHitsUpTo(10000);
        source.fetchSource(new String[]{"spu_id","title","price","main_image","popularity_score"},
                         new String[]{"detail_content","marketing_ext"});
        source.sort(new FieldSortBuilder("popularity_score").order(SortOrder.DESC));
        source.sort(new FieldSortBuilder("spu_id").order(SortOrder.ASC));
        if (request.getSearchAfterToken() != null) {
            source.searchAfter(SearchAfterTokenCodec.decode(request.getSearchAfterToken()));
        }
        SearchRequest esRequest = new SearchRequest(resolveIndex(request));
        esRequest.source(source);
        SearchResponse response = client.search(esRequest, RequestOptions.DEFAULT);
        return ProductSearchConverter.toPage(response);
    }
    // buildQuery() builds a BoolQuery with filter clauses and a multi_match for the keyword.
}

Key points: trackTotalHitsUpTo(10000) avoids expensive total‑hit counting.

Sorting includes a tie‑breaker ( spu_id) to guarantee deterministic order.

Only necessary fields are fetched, reducing fetch‑phase cost.

Write‑Path Optimizations

Adjust refresh_interval to 30 s during heavy ingestion, disable ( -1) for bulk reindex, then manually refresh.

Perform full reindex on a new index (e.g., product_v20260511), load data, backfill increments, warm up queries, then switch alias atomically.

Use external versioning ( VersionType.EXTERNAL_GTE) for idempotent writes and to avoid out‑of‑order updates.

Configure bulk processor with exponential backoff, dead‑letter queue, and alerting on partial failures.

Idempotent Index Request

public void indexProduct(ProductIndexMessage msg) {
    IndexRequest req = new IndexRequest("product_search")
        .id(msg.getSpuId())
        .source(msg.getDocument(), XContentType.JSON)
        .versionType(VersionType.EXTERNAL_GTE)
        .version(msg.getVersion());
    bulkProcessor.add(req);
}

JVM and Cluster Tuning – Last‑Mile Optimizations

Heap limited to 30 GB on 64 GB nodes (‑Xms30g ‑Xmx30g) with G1GC; reserve 25 % for OS page cache.

Disable swap and lock memory ( bootstrap.memory_lock: true, vm.swappiness=1).

Search thread‑pool queue size tuned to 2000, write queue to 500; always paired with request‑level rate‑limiting and circuit‑breakers.

Merge policy tuned ( max_merged_segment=5gb, segments_per_tier=12, max_thread_count=1) to avoid large segment creation and I/O storms.

Results – 5× Throughput Gain

After the full set of changes, the same hardware delivered:

Peak QPS: 20 000 → 100 000+

P99 latency: 3 200 ms → 180 ms

Hot‑node CPU: 92 % → 55 %

Full GC frequency: 8‑12 times/hr → ≈0

Search‑thread rejections: 500+/min → 0

Write latency: 45 min → ≤3 s

Most impactful actions (in order of impact):

Eliminate deep pagination and dangerous DSL.

Introduce a query‑governor with throttling and degradation.

Asynchronous, idempotent bulk writes.

Converge the index model and disable dynamic mapping.

Hot‑warm‑cold tiering and balanced shard sizing.

Fine‑tune JVM, merge policy, and thread pools.

Practical Checklist for Billion‑Scale Elasticsearch Deployments

Query Side

Any from + size deep pagination allowed?

Wildcard‑only queries present?

Frequent script_score usage?

Heavy aggregations on the front‑end?

Requests bypassing an application‑level guard?

Index Side

Dynamic mapping enabled?

Field explosion or cluster‑state bloat?

Oversized _source fields?

Imbalanced shard sizes?

Hot and cold data mixed in the same index?

Write Side

Refresh interval too aggressive?

Bulk parameters uncontrolled (size, concurrent requests)?

Lack of idempotency or retry logic?

Full reindex performed on the live index?

Engineering Side

Cache, rate‑limit, circuit‑break, degradation in place?

DSL fingerprinting and slow‑query attribution?

Capacity model and load‑test baseline established?

Fast alias switch‑over and rollback procedure?

If more than half of these items apply, the cluster likely still needs a production‑grade governance overhaul.

Future Directions – From “Survivable” to “Sustainable”

Search‑Recommendation Fusion : Add sparse and vector recall, hotness ranking, and rule‑based re‑ranking as multi‑stage pipelines.

Independent Search Platform : Extract query rewrite, recall orchestration, rule engine, cache, and observability into a dedicated service layer.

Full Read‑Write Separation & Multi‑Cluster Governance : Split by business domain, hot‑cold data, and write vs. query workloads; route via a unified search gateway.

Overall, the 5× performance boost was achieved by treating the search stack as a whole system—shielding Elasticsearch from harmful traffic, aligning index design with query patterns, separating write and read resources, and applying targeted JVM and merge‑policy tuning.