Youzan Search Engine Practice – Engineering Part: Architecture, Indexing, and Performance Optimization

The rapid growth of e‑commerce data creates challenges for extracting useful information from massive historical and real‑time datasets. Youzan, a medium‑size e‑commerce platform, processes millions of raw records and billions of user behavior events daily, typically stored in three systems: relational databases (MySQL), Hadoop for large‑scale logs, and search engines such as Elasticsearch or Solr.

To achieve commercial‑grade search quality, Youzan built a distributed real‑time search engine based on Elasticsearch, which itself relies on the Lucene index library. The system integrates seamlessly with MySQL and Hadoop, and adds a custom advanced search module that provides flexible relevance calculation and plugin‑based extensions.

Index construction follows a queue‑driven architecture: data is first written to a database or file, then synchronized to Kafka (using tools like MyPipe or Alibaba Canal). Elasticsearch subscribes to relevant Kafka topics for real‑time indexing, while full‑batch indexing is performed via a Hadoop‑ES pipeline that treats the index as a Hive external table, enabling transparent distributed indexing.

Advanced Search (AS) serves as a proxy layer, offering distributed sharding, fault‑tolerance, a plugin framework for custom relevance algorithms, and a query cache to prevent load spikes. Plugins can implement query rewriting, re‑ranking, and other business‑specific logic, keeping the core engine extensible.

Performance optimizations include an application‑level queue to prevent avalanche effects during traffic spikes, automatic degradation to simpler queries when response times exceed thresholds, and extensive use of Lucene filters (bitsets) to accelerate query execution. Additional tuning advice covers disabling automatic shard rebalancing, extending refresh intervals, removing the default "_all" field, and physically separating slow and fast indexes.

The article concludes with a summary of the architecture, indexing mechanisms, advanced search capabilities, and practical optimization tips, aiming to provide general guidance for building commercial e‑commerce search engines.