Intelligent Operations for Tencent Cloud Big Data Platform: Challenges, Practices, and Future Directions

On December 15, Tencent Cloud hosted the first "Tencent Cloud+ Community Developer Conference" in Beijing, gathering more than 40 technical experts to discuss the latest developments in AI, big data, IoT, mini‑programs, and operations development. The following is a structured summary of the big‑data track.

The Tencent Cloud big‑data platform faces typical characteristics of large‑scale clusters: massive data volumes, many components, and complex inter‑module interactions. Rapid problem discovery, resolution, and optimization for diverse customer scenarios are critical topics.

Current Product Landscape

Tencent Cloud offers a full‑stack big‑data matrix, including TBDS (the core platform refined on hundreds of PB of internal data), Sparkling (a fully managed, PB‑scale data‑warehouse), Snova (Mpp‑style data warehouse), EMR (managed Hadoop), SCS (Flink‑based stream computing), Elasticsearch Service, and various BI and search services. These services can be combined with databases, message queues, and object storage to build flexible big‑data applications.

Challenges

Key challenges include the inherent complexity of distributed clusters, diverse customer workloads (e‑commerce, finance, gaming), and the need for near‑real‑time performance, which makes fast issue detection and resolution difficult.

Intelligent Operations (AIOps) Concept

AIOps integrates big‑data platforms with AI algorithms to form a closed‑loop: data collection → analysis & decision → automated remediation. Traditional rule‑based automation cannot keep up with the scale and variability of modern workloads, so machine‑learning‑driven decision making is introduced.

Best Practices – Exploration

Logs and metrics are aggregated using Elasticsearch, Jaeger (for RPC tracing), and dr‑elephant (for Hadoop/Spark job profiling). These tools provide a unified view of system health, enabling rapid root‑cause analysis.

Best Practices – Detection

Statistical methods (3‑sigma, control charts, EWMA, weighted moving average) are applied for threshold‑free anomaly detection. Unsupervised techniques such as one‑class SVM and Isolation Forest are used to model normal behavior and flag deviations. Supervised models (logistic regression, LightGBM, deep neural networks) are trained on labeled anomalies to improve detection accuracy.

Best Practices – Optimization

Parameter tuning for EMR/Hadoop jobs is addressed with regression models and reinforcement learning (Q‑learning, DQN, Double DQN). These approaches predict performance impact of configuration changes and iteratively improve resource utilization.

System Architecture

The solution consists of an offline pipeline (Kafka → HDFS → Spark for feature extraction and model training, models stored in COS) and an online pipeline (Kafka → Flink for real‑time preprocessing, feature engineering, model inference, and CEP‑based rule execution). Detected anomalies trigger alerts via cloud functions (WeChat, SMS, email) and can initiate automatic scaling actions.

Experimental results show >88% precision and ~80% recall for the anomaly detection system. Future work includes clustering time‑series for model selection, long‑term pattern analysis, and transferring learned models across business domains.

Future Outlook

The roadmap focuses on four directions: scenario‑driven root‑cause analysis and automated recovery, productization of internal alerts and tuning tools, platformization to accelerate algorithm integration, and knowledge‑driven model reuse across different workloads.