Tencent's Autonomous Big Data Platform: Data‑Driven Governance and AI‑Powered Optimization

In recent years, the rapid growth of big data workloads has created new operational challenges, including the need for unified governance models and tools to continuously optimize task efficiency and stability. Tencent’s big data platform addresses these issues with a data‑plus‑algorithm driven “platform brain” that enables one‑stop self‑diagnosis, self‑optimization, and self‑management, lowering the barrier for using big data products.

The article outlines four main sections: trends and challenges, the autonomous solution, practical implementations, and a summary outlook.

Trends and Challenges : The big data technology stack has evolved from data warehouses to AI, leading to exponential growth in task volume and complexity. Traditional expert‑driven approaches can no longer sustain the scale, creating a “triple constraint” of quality, cost, and efficiency.

Autonomous Solution : Tencent’s autonomous approach, called the Platform Brain, builds from an observable data foundation upward to decision‑making capabilities, progressing through semi‑automatic to fully automated intelligence. Three key capabilities are required: (1) an observable data foundation that captures task, data, and service metrics; (2) a pluggable decision engine that can adapt to component changes; and (3) generalized decision intelligence that combines rule‑based, supervised, optimization, planning, and reinforcement‑learning methods.

Architecture : The solution is organized into three layers—perception, analysis, and decision. The perception layer standardizes data collection and creates a unified metric store. The analysis layer transforms these metrics into knowledge bases and integrates machine‑learning models. The decision layer delivers services such as full‑link diagnostics, health scoring, cluster inspection, and automated tuning.

Practical Practices : Real‑world implementations include:

Spark parameter tuning using both rule‑based and black‑box (ML) methods, achieving 30‑70% resource reduction and 20‑45% performance gains.

JVM parameter tuning focused on GC settings, combining expert rules with automated search to improve tasks with GC bottlenecks.

SQL engine selection (Presto vs. Spark) driven by historical rules and enhanced by machine‑learning models, reducing failure rates by 60‑80% and achieving >90% success.

Key deployment challenges involve balancing optimization quality against computational cost, ensuring fault‑tolerance, and maintaining timely data for decision making.

Summary and Outlook : The autonomous big data platform leverages data, expert knowledge, and AI to improve problem detection, root‑cause analysis, fault remediation, and resource optimization. Decision‑intelligence solutions evolve from rules to planning and reinforcement learning, with pluggable architectures enabling rapid adaptation to new scenarios. Continued focus on the three AI pillars—data, algorithms, and compute—will drive the next generation of intelligent data products.