Why Caching Is the Secret Weapon for High‑Performance Search Engines

Interesting Search Facts

Query length: most user queries are short, typically 3‑7 characters.

Query characteristics: 30‑40% of queries are repeated; queries exhibit temporal locality.

Query distribution: about 64% of queries appear only once ("isolated queries"), while the top 25 hot queries account for 1.2‑1.5% of total traffic.

Query habit: ~64% of users only view the first result page (top‑10), 12% view the second page (top‑20), and fewer than 12% look beyond the third page.

These statistics, although dated, consistently show a power‑law distribution where roughly 20% of query terms generate 80% of the traffic.

Simple Search System Framework

A typical search system can be divided into five layers:

Search entry layer : the CGI interface that receives user requests, enriches them with user profile data, and forwards them to the backend.

Integration & ranking layer : modules such as qp (query processing), proxy, mixer and rank perform query rewriting, traffic balancing, doc assembly, filtering, and multi‑stage ranking.

Retrieval & recall layer : interacts with index clusters (fresh and full) to recall documents and perform set intersections.

Data layer : handles raw data ingestion, vector computation, feature extraction, and maintains up‑to‑date searchable indexes.

Operations layer : provides monitoring, logging, experiment analysis, and other tools to ensure system health.

During a query flow, data moves from the entry layer through integration, retrieval, and ranking before the final result is returned to the client.

Different Cache Types

Result‑level cache : stores the final search result list; a cache hit returns the result instantly without any further computation.

Intermediate‑value cache : caches auxiliary data such as user profiles, qp strings, inverted index postings, or partial intersections, reducing the amount of work needed in later stages.

Multi‑layer mixed cache : combines result‑level and intermediate caches across layers; if the top layer misses, lower layers can still provide cached intermediate data to accelerate processing.

Cache Strategies

Update & eviction policies : dynamic strategies (FIFO, LFU, LRU, TTL, etc.) adapt to real‑time traffic, while static strategies pre‑warm caches for known hot events (e.g., holidays).

Prefetch strategies : predict user behavior to pull data early, such as fetching user parameters while the user types or loading the next page’s results in advance.

Local vs. distributed cache : local cache offers ultra‑low latency within the same process but lacks sharing; distributed cache (Redis, Memcached, etc.) provides shared storage at the cost of network overhead.

Potential Problems of Caching

Long‑tail queries remain slow because caches mainly benefit hot queries.

Staleness and consistency issues arise when the underlying database updates but the cache does not.

Classic cache pitfalls: penetration (non‑existent data), breakdown (hot key expiry), avalanche (mass expiry), and cold‑start (cache warm‑up).

Increased resource consumption and maintenance complexity.

Cache‑key design directly impacts hit rate; poor keys lead to low effectiveness.

Cache object size can cause serialization overhead and network latency.

Conclusion

Cache is a powerful tool to trade space for time, improving average query latency and reducing backend load, but it does not replace algorithmic optimization. Properly balancing efficiency and effectiveness requires careful selection of cache granularity, strategies, and continuous monitoring of hit rates and latency distribution.