Tencent's Elasticsearch Practices: Application Scenarios, Challenges, Optimizations, and Future Directions

Elasticsearch (ES) is a popular open‑source distributed search and analytics engine that Tencent uses extensively for real‑time log analysis, full‑text search, and structured data queries, dramatically reducing the cost of extracting value from data.

Within Tencent, ES supports large‑scale internal scenarios such as operational logs, business logs, and audit logs, offering features like a complete Elastic ecosystem, sub‑second latency, flexible search capabilities, and rapid response even at trillion‑level log volumes.

External industry use cases include e‑commerce product search, app store search, and site‑wide search, where ES delivers high performance (up to 100k+ QPS) and strong relevance while maintaining four‑nine availability and disaster‑recovery capabilities.

Time‑series scenarios (metrics, APM, sensor data) demand extremely high write throughput (up to 10 million writes per second) and low storage costs; challenges focus on storage and compute efficiency.

Challenges encountered are divided into two categories: (1) Search‑oriented challenges—high availability and high performance under massive query loads; (2) Time‑series challenges—cost‑effective storage, high‑concurrency writes, and multi‑dimensional analysis.

To address these, Tencent implements a three‑dimensional high‑availability strategy: system robustness, disaster‑recovery solutions, and fixing system defects. Optimizations include service throttling, metadata control improvements (10× cluster expansion), shard‑balancing, and memory‑level flow control.

Cost‑optimization techniques involve hot‑cold data separation, pre‑computation (Rollup), archival to cheap storage, and cache‑based memory savings, reducing storage costs and improving query performance.

Performance enhancements cover write‑path optimizations (45 % faster primary‑key deduplication, vectorized execution), CPU utilization improvements (20 % faster translog refresh), and query optimizations (segment pruning, CBO, hardware acceleration).

Merge‑strategy refinements introduce time‑aware merging for time‑series data and automatic cold‑data merging, cutting unnecessary segment scans and boosting search performance.

These improvements have yielded measurable benefits for customers such as a major e‑commerce platform, including higher reliability, robust disaster recovery, and increased operational efficiency.

Looking forward, Tencent contributes its enhancements back to the open‑source community, has submitted over ten pull requests to Elastic, and plans to further explore online services and OLAP analytics built on ES, while continuing to refine large‑scale query handling and index management.