From Zero to One: Building and Optimizing Search Engines with Elasticsearch – Insights and Case Studies

The talk begins with an overview of the essential components of a complete search engine and recommends starting with open‑source solutions such as Sphinx, Lucene/Solr, or Elasticsearch, highlighting Elasticsearch’s distributed nature, JSON document storage, rich plugin ecosystem, and built‑in X‑Pack machine‑learning features.

It then describes the process of building a search system: selecting the appropriate engine, handling data import/export from big‑data pipelines, implementing real‑time synchronization of inserts, updates, and deletions, and exposing the service via middleware that performs result ranking, ad insertion, and user‑behavior logging.

The article explains Elasticsearch’s read/write model built on Lucene, detailing immutable segment files, the translog for durability, refresh operations that make new data visible, and flush operations that persist data to disk.

Shard distribution and coordination are covered, showing how queries are routed to relevant shards, aggregated, and sorted, with guidance on choosing shard counts based on read‑heavy or write‑heavy workloads.

Cache mechanisms such as query cache, segment caching, and OS‑level mmap are discussed, along with best practices to avoid expensive range queries, over‑sharding, and unnecessary index updates.

Several practical case studies are presented: a sudden query‑cache explosion caused by large range queries and its mitigation using script‑based filtering; an isolation failure during a Double‑11 event resolved by grouping nodes and limiting shared‑cluster impact; and scan‑induced memory pressure solved by replacing scan with search_after.

Deployment experiences at Youzan are shared, including configuration tweaks for garbage‑collection, transparent huge pages, node‑level routing allocation, index templates, and strict mapping to prevent type‑inference errors.

The overall system architecture is outlined: a business layer exposing RPC or REST APIs, an application layer handling query parsing, rewriting, re‑ranking, and logging, a caching layer that stores query results, and a data layer that integrates Elasticsearch with ETL tools like DataX, all running on a mix of cloud and dedicated servers.

Specific functional examples cover product search with multi‑tag arrays, order search with hot‑cold data segregation, and cluster management using monitoring plugins, highlighting Elasticsearch’s limited auto‑balancing and manual strategies for disk‑based rebalancing and data migration across clusters.