Distributed Search Engine Design and Index Management in WeChat Search

Introduction: The article discusses the challenges of designing a good distributed system for search, noting that many practitioners understand theories like Paxos and CAP but lack concrete design experience. The author, with eight years of experience in distributed storage and search systems, presents the evolution of WeChat Search's distributed architecture.

Background: Search engines such as Baidu, Google, and Elasticsearch rely on inverted indexes. Elasticsearch uses a peer-to-peer architecture where each node handles both writes and reads, making it suitable for workloads that are not latency‑critical. In large‑scale systems, indexing is often off‑loaded to offline processes to avoid impacting write performance.

Data Sharding: The core of distributed problem solving is splitting large tasks into smaller ones. In search systems, data is divided into shards (called shards in ES, regions in HBase, tablets in Bigtable). Sharding is distinguished from replication (copies of shards). Sharding can be performed by key range or hash; hash is more common to avoid data skew. Rebalancing moves shards between nodes when scaling.

System Design Considerations: A leader (selected via Chubby) manages shard‑to‑node mappings, node status, and routing. The leader does not participate in online search requests, keeping control flow separate from data flow. Heartbeats from nodes allow the leader to monitor health and assign tasks such as index loading.

Index Management: The architecture adopts a Lambda‑style pipeline with both batch (offline) and stream (near‑real‑time) processing. Batch indexing rebuilds the full index periodically, while near‑real‑time updates use an LSM‑like structure (Refresh and Level libraries) to provide low‑latency writes. Full‑index rebuilds are triggered by operators or timers, and the leader coordinates creation of new index libraries, loading them into Searcher groups, and retiring old libraries.

Shard‑to‑Group Mapping: Searcher nodes are organized into groups; each group loads a subset of shards. When scaling out, new groups are added and shards are rebalanced accordingly. The mapping evolves from shard‑to‑group to shard‑to‑full‑index‑library to support seamless upgrades.

Transaction and Consistency: The system treats control operations (e.g., scaling, index rebuild) as low‑frequency, single‑node tasks managed by the leader, while data flow remains highly available. Consistency requirements are relaxed; most search workloads only need monotonic read consistency, achieved by routing a user’s queries to the same node.

Conclusion: The article summarizes the trade‑offs made in WeChat Search’s distributed design, covering leader election, sharding, index lifecycle, and the blend of Lambda and Kappa architectures. It highlights practical lessons for building scalable, reliable search services on top of distributed storage systems.