Operational Challenges and Strategies for Tencent Cloud Redis

Author: Feng Weiyuan, senior engineer and head of Tencent Cloud Redis operations, with six years of DBA experience covering SQL optimization, instance tuning, database architecture, massive cluster operations, and platform management for services such as QQ, Qzone, QQ Music, Weiyun, and Tencent Cloud.

Tencent Cloud Redis, launched in 2015, has grown explosively to serve tens of thousands of customers. The author, as the sole operations lead, faces three major challenges: metadata consistency management, efficient operation of ten‑thousand‑device fleets, and intelligent scheduling.

Overview of Tencent Cloud Redis

Tencent Cloud Redis is built on years of distributed cache expertise from QQ, Music, Qzone, and Weiyun. It offers a high‑availability, high‑reliability Redis service platform with over ten thousand devices and QPS reaching the hundred‑million level. Three versions are available – master‑slave, cluster, and a next‑generation edition – all compatible with the Redis protocol and supporting strings, lists, sets, sorted sets, and hashes. Features include master‑slave hot‑standby, automatic disaster recovery, data backup, fault migration, instance monitoring, online scaling, and data rollback.

Operational Issues Encountered

During Redis operation, the following problems are commonly observed:

Environment: network and TCP parameter settings.

Design: page‑table replication causing latency during persistence.

Developers: slow queries, connection storms, lack of flow control.

End users: traffic spikes (e.g., e‑commerce flash sales) that push processing to its limits.

Overall, the service suffers from a mismatch between resource demand and supply.

Three Major Challenges

Challenge 1 – Metadata Consistency Management

Inconsistent metadata leads to frequent operational failures. The four core metadata types are cluster, device, instance, and configuration. Three principles are applied:

“Full” – comprehensive inventory of all metadata.

“Accurate” – alignment with the live network.

“Unified” – a single entry point with unified APIs for reading and modifying metadata, enabling auditability.

The solution involves cataloguing all metadata, extracting common attributes, modeling them, defining template objects, and exposing APIs. This creates a DB‑CMDB subsystem that provides unified metadata management across databases.

Challenge 2 – Efficient Operation of Ten‑Thousand Devices

Manual operations become impossible when the customer base explodes, reaching billions of QPS. A “Job Platform” was built to host operational logic, based on three pillars:

Platformization – atomic tools hosted on the platform.

Process‑ization – tools chained into reusable workflows.

Visualization – clear, visual representation of operations.

Scripts are treated as atomic tools with only success or failure outcomes. Tools are combined into workflows that can be reused for tasks such as machine provisioning, Redis migration, and scaling. The platform now supports hundreds of scenario‑based workflows, handling thousands of daily calls, reducing incident rates, and improving operational efficiency by 300%.

Challenge 3 – Intelligent Scheduling

Manual triggers only achieve semi‑automation. Two systems are introduced:

Automated Scheduling System – triggers based on time (e.g., weekly restarts) or events (e.g., alerts). Events are registered, captured, and invoke jobs or workflows, forming a closed‑loop operation.

Decision System – gathers information before actions, applies decision trees or AI models, and determines whether and which workflows to invoke.

These systems enable fully automated, event‑driven operations.

Operational Maturity and DBA Requirements in the Cloud Era

Maturity progresses from primitive manual methods to standardized tools, then to visualization, process‑driven platforms, and finally to automated, event‑driven scheduling with AI‑assisted decision making. DBA responsibilities now extend beyond stability to product design, architecture, component source code, community engagement, and personal influence. DBAs must act as operators, developers, and product owners, embodying the “service‑oriented, DevOps‑integrated” trend.

Additional Resources

Recommended readings include articles on senior coding, Tencent Cloud’s native DevOps, cloud computing fundamentals, and career progression for programmers.