Tencent Optimizes Elasticsearch High-Concurrency Write Performance, Cutting 10M Data Load Time by 20%

Recently, Shay Banon, the founder and CEO of Elasticsearch, sent a special email to Tencent Cloud to thank the Tencent team for their contributions to the Elasticsearch open‑source community.

Tencent engineers submitted code that successfully optimized Elasticsearch's high‑concurrency write performance, reducing the write time for 10 million records from 18 minutes to 15 minutes – a 20% speed increase – and shared the solution with the community.

In his thank‑you letter to Tencent’s Vice President and Cloud President Qiu Yuepeng, Banon wrote that the improvement, proposed by a Tencent developer, increased write speed for certain Elasticsearch types by about 20% and expressed eagerness for continued cooperation.

Earlier this year, at the Tencent Global Digital Ecology Conference, Tencent Cloud and Elastic signed a strategic cooperation agreement covering product, technology, and ecosystem collaboration. Leveraging Elastic’s search software strengths and Tencent Cloud’s cloud‑computing and big‑data capabilities, they launched the Tencent Cloud Elasticsearch Service with advanced features such as cluster permission management, SQL, machine learning, and graph queries.

The optimization originated from a real‑world scenario in Tencent’s internal one‑stop intelligent big‑data service platform, “National BI”. The platform’s statistical functions rely on Elasticsearch’s open‑source code. Daily full‑data imports of billions of records and over 500 tags created a bottleneck, taking several hours despite multiple parameter tweaks.

Analyzing Elasticsearch’s write mechanism, the team discovered that many threads were blocked by writeLock held by rollGeneration, causing long lock‑wait times. They shifted focus from parameter tuning to thread optimization, ultimately improving rollGeneration to cut the 10‑million‑record write time from 18 minutes to 15 minutes.

The team expressed gratitude to Elastic co‑founder and technical lead Simon Willnauer for his support, noting that the optimization greatly enhanced the National BI system’s data analysis efficiency and that the new features are now available on Tencent Cloud Elasticsearch Service.

This achievement is just the beginning; further optimization opportunities remain, and Tencent Cloud is actively communicating with the open‑source community.

Since 2010, Tencent has pursued an open‑source strategy, moving from isolated development to collaborative governance, contributing code to projects such as OpenStack, Linux Foundation, LFAI, MariaDB, and Apache foundations.

In recent years, Tencent Cloud contributed 40 patches to the KVM kernel, ranking on the global open‑source contribution list for two consecutive years, and released four core infrastructure projects to the OCP community.

By August 2019, Tencent had published 82 open‑source projects on GitHub across areas like WeChat, cloud, gaming, AI, and security, accumulating over 240,000 stars and wide recognition.

Looking ahead, Tencent plans to broaden its open‑source scope, strengthening technical capabilities through open, shared, and collaborative development models.