Building a Production-Grade Search Engine with Spring Boot & Elasticsearch

When to Introduce Elasticsearch

Relational databases handle point lookups and simple transactions well, but their B+Tree indexes break down when search dimensions multiply. A LIKE '%keyword%' forces a full table scan; with five or six WHERE conditions, composite index hit rates plummet, and filesort plus temporary tables appear frequently. MySQL also lacks BM25/TF-IDF relevance scoring, so result order is arbitrary. Aggregations with GROUP BY plus COUNT/SUM on tens of millions of rows can exhaust memory.

Three hard thresholds signal it's time to adopt ES:

Single-table data exceeds ~5 million rows, complex query latency consistently exceeds 2 seconds, and adding indexes yields diminishing returns.

Business requires full-text search, synonym expansion, geo-fencing, multi-tag cross-filtering, or real-time aggregation drill-down.

Read/write ratio is heavily skewed (9:1 or higher); ES's inverted-index architecture is built for read-heavy loads.

Bottom line: ES is a search and aggregation engine, not a transactional database. From day one you must accept eventual consistency , trading latency for throughput and using compensation mechanisms instead of strong consistency. Do not expect it to handle high-concurrency writes while guaranteeing ACID.

Data Synchronization: Stop Debating Between Dual-Write and Canal

Moving data from MySQL to ES is the first implementation hurdle. Three common patterns exist; the choice depends on team ops capability and business tolerance.

Application dual-write — simplest: call MySQL and ES APIs together. Lowest latency but couples business logic tightly; an ES hiccup or network timeout stalls the main flow. Suitable only for early validation.

Logstash JDBC polling — scheduled SELECT for incremental data. Easy to configure, but minute-level latency cannot support real-time search; typically used for historical migration or offline reports.

Canal + Kafka + consumer — the production mainstream. Canal masquerades as a MySQL slave to parse binlog, pushes change events to Kafka, and a Spring Boot consumer writes to ES. Zero business-code intrusion, supports resume-from-checkpoint, and full/initial sync can run in separate consumer groups. Kafka absorbs write spikes while ES consumes at its own pace.

Critical deployment details:

Enable Canal GTID mode so master failover does not lose offsets.

Disable Kafka auto-commit; manually ACK after processing. Do not chase Kafka's theoretical Exactly-Once; production relies on manual ACK + ES primary-key idempotent upsert to achieve eventual consistency.

Separate full-sync and incremental-sync consumers into different groups; mixing them causes offset-commit conflicts.

graph LR
A[MySQL] -->|Binlog| B(Canal Server)
B -->|JSON Event| C{Kafka Topic}
C --> D[Consumer A: Historical Full Load]
C --> E[Consumer B: Real-time Incremental Sync]
E --> F[Spring Boot ES Service]

Mapping Design: Get It Wrong and Reindexing Is Painful

ES performance hinges on index design. Once mapping is set, changing analyzers or adding fields usually requires a full reindex — costly at scale.

Don't use the default analyzer. For Chinese, install the ik plugin. Production typically configures two strategies: ik_max_word for fine-grained recall and ik_smart for coarse-grained display. Attach synonym and stopword filters via a filter chain; place synonyms.txt in the config directory and hot-reload with _reload_search_analyzers without downtime. Filtering stopwords like "的、了、在" shrinks the inverted index significantly and speeds queries.

Choose field types deliberately. Tokenized search → text; exact match, sort, aggregation → keyword. Complex fields use multi_fields to store both. Pure-display fields (e.g., rich-text body) set index: false to skip inverted index entirely. doc_values (on by default) powers sorting/aggregation; disabling it for fields that never sort or aggregate saves ~30% heap. object type flattens nested structures, causing cross-condition pollution; use nested to preserve hierarchy, but note each nested document creates a separate inverted entry — write amplification is real, so avoid overuse.

Size shards and refresh interval to data volume. ES 7+ sweet spot: 20–50 GB per shard. For 500 GB, 10–25 primary shards suffice. Avoid a single shard swallowing all writes, and avoid dozens of shards overwhelming the coordinating node. Default refresh_interval: 1s is too aggressive for non-real-time workloads; raising to 30s drastically reduces segment merge pressure and visibly boosts write throughput. Enable "codec": "best_compression" (DEFLATE) for read-heavy scenarios — saves ~40% disk with acceptable CPU overhead.

Spring Data ES Query Encapsulation: Stop Concatenating JSON Strings

Since Spring Data ES 4.x, TransportClient is retired; the stack standardizes on ElasticsearchRestTemplate and ElasticsearchOperations. Production queries must use strongly-typed builders — string concatenation is unmaintainable and prone to escaping bugs.

@Service
@RequiredArgsConstructor
public class SearchFacade {
    private final ElasticsearchOperations esOps;

    public PageResult<Article> search(ArticleSearchReq req) {
        BoolQueryBuilder bool = QueryBuilders.boolQuery();

        // 1. Full-text match goes into scoring context
        if (StringUtils.isNotBlank(req.getKeyword())) {
            bool.must(QueryBuilders.matchQuery(
                "title", req.getKeyword())
                .operator(Operator.AND));
        }

        // 2. Deterministic filters go into filter context (no scoring, cacheable)
        bool.filter(QueryBuilders.termsQuery("status", List.of(1, 2)));
        if (req.getCategory() != null) {
            bool.filter(QueryBuilders.termQuery("category", req.getCategory()));
        }
        if (req.getStartTime() != null) {
            bool.filter(QueryBuilders.rangeQuery("publish_time").gte(req.getStartTime()));
        }

        NativeQuery query = NativeQuery.builder()
                .withQuery(bool)
                .withPageable(PageRequest.of(req.getPage(), req.getSize()))
                .withHighlight(highlight -> highlight
                        .fields(hf -> hf.field("title").preTags("<em>").postTags("</em>")))
                .build();

        SearchHits<Article> hits = esOps.search(query, Article.class);
        return convertToPageResult(hits);
    }
}

Battle-tested habits:

Never mix must and filter. Status, category, time ranges — all deterministic — go into filter; ES places them in the Filter Cache, skipping scoring, yielding extreme speed. Only genuine text matching belongs in must.

Enable highlighting only on text fields. Frontend renders <em> tags directly — cleaner than server-side string stitching.

For aggregation drill-down, terms aggregation defaults to top 10 buckets. If the business needs the full distribution, increase size or use composite aggregation with cursor pagination; otherwise large datasets silently truncate, misleading downstream consumers.

Deep Pagination & Performance Tuning: Lessons from Production Scars

from + size

beyond 10,000 is rejected by ES. Don't fight it by raising max_result_window; change the approach.

Backend exports / offline batch jobs: Use the Scroll API. It snapshots the current result set and maintains a server-side cursor. Remember to call clear_scroll to release resources. Do not use Scroll for customer-facing list pagination — snapshots hide concurrent updates and add latency.

App lists / infinite scroll: Use Search After. Stateless cursor; the sort values of the last document on the previous page locate the next page. Sort fields must be unique to avoid skipped or duplicate results. A composite sort of publish_time desc + _id desc is usually sufficient.

Sort sort = Sort.by("publish_time").descending().and(Sort.by("_id").descending());
Object[] searchAfter = new Object[]{lastTime, lastId};
NativeQuery nextQuery = NativeQuery.builder()
        .withQuery(bool)
        .withSorts(sort)
        .withSearchAfter(searchAfter)
        .withPageable(PageRequest.ofSize(req.getSize()))
        .build();

Performance tuning focuses on a few levers:

Routing pre-filter: In multi-tenant systems, isolate data by tenant_id. Pass routing=tenant_id on both writes and reads; queries hit only the relevant shard, cutting scanned data by ~90% and collapsing latency.

Avoid Script queries: Painless scripts run in heap memory and easily trigger long GC. Pre-compute derived fields in the sync pipeline and persist them.

Watch thread pools religiously: Monitor thread_pool.search.active and queue length. A saturated queue means shard compute is overwhelmed. Adding data nodes won't help; first hunt for massive aggregations or unfiltered full-index term queries, optimize those, then decide on scaling.

Async Sync Consistency: Rely on Safety Nets, Not Luck

Binlog async replication to ES inevitably loses events or suffers network blips. Systems that run stably never gamble; they build mechanisms to catch failures.

Reconciliation job is the last line of defense. Daily off-peak job pulls MySQL primary keys changed in the last 7 days, batch-fetches version numbers or field hashes from ES, and republishes mismatches to Kafka for replay — with full audit logging. This single mechanism catches 99% of sync anomalies.

Dead-letter queue (DLQ) with exponential backoff. Consumer parse failures or ES write failures must not be swallowed. Divert exceptions to a dedicated DLQ Topic or retry table. Retry intervals: 1s → 5s → 30s → 5min. After 5+ failures, escalate to a manual ticket. Consumers must be idempotent; ES native _id upsert is naturally idempotent, but under concurrency use MySQL's update_time to order versions and prevent stale overwrites.

Handle version conflicts gracefully. ES 8.x deprecated _version in favor of seq_no + primary_term for optimistic concurrency. Include both on writes:

IndexRequest request = new IndexRequest("articles");
request.id(article.getId().toString())
        .source(json)
        .setIfSeqNo(currentDoc.seqNo())
        .setIfPrimaryTerm(currentDoc.primaryTerm());

A 409 Conflict signals concurrent modification. Strategy: treat MySQL as source of truth, ignore the conflict, fetch the latest seq_no, and retry. Application layer catches VersionConflictEngineException, merges old and new fields, then rewrites — safer than blind overwrite.

Production Pitfalls & Evolution Roadmap

Split-brain is virtually eliminated in 7+ via Voting Config Exclusions. Just ensure discovery.seed_hosts lists all master-eligible node IPs — no single points.

Frequent Full GC / pauses: 90% caused by oversized heap. Cap ES JVM heap at 30 GB; beyond that compressed oops disable, slowing allocation. For aggregations with many buckets, force composite aggregation cursor pagination instead of one-shot full returns. Enable G1GC with -Xms = -Xmx to keep pauses under 100 ms.

Slow query log spam: usually misused filters or regex abuse. Set index.search.slowlog.threshold.query.warn to 500 ms; periodically inspect stages with _profile API. Avoid regex where possible; preprocess into wildcard or dedicated inverted fields.

Write OOM: bulk request body too large. Keep single bulk at 5–15 MB or 1,000–5,000 docs. For non-real-time sync, increase refresh_interval and stop fighting segment merges.

Architectural evolution, not big-bang. Start with 1 master + 2 data nodes; get full/incremental sync working for MVP search. As data grows, adopt ILM (Index Lifecycle Management) to tier cold data (logs, historical orders) onto Warm nodes (cheap HDD), keeping Hot nodes (SSD) for the last 30 days — slashes hardware cost. For cross-DC HA, add CCR (Cross-Cluster Replication) for read-only replicas or use ECK for cloud-native orchestration. When LLM workloads arrive, reserve dense_vector fields and plug in knn for semantic recall, gradually migrating toward a unified "keyword + vector" hybrid search gateway.

Closing Thoughts

ES is no silver bullet — it consumes both performance and design discipline. A single mapping mistake can cost days of reindexing; an unguarded sync pipeline invites daily data-mismatch fire drills; poor query habits make cluster scaling futile against tail latency.

Landing this engine is not about tweaking a few APIs. It's about stitching the sync pipeline, query standards, observability (Prometheus scraping _nodes/stats and slow logs), and reconciliation into a closed loop. Early on, enforce mapping changes through code review; extract search into a dedicated middleware service, decoupled from business code. Build observability and compensation first — only then does ES graduate from "slow-query stand-in" to "business-growth foundation".