Why Smarter Data Platforms Are Harder to Operate and How Next‑Gen System Intelligence Solves It

Background: Growing Complexity with Smarter Platforms

Computation intelligence makes agents "run fast", storage intelligence lets agents "store and retrieve", and data intelligence enables agents to "analyze for you". While these capabilities improve service, they also make platform operations harder because workloads have become multi‑modal (batch ETL, model training, online inference, RAG retrieval, Agent orchestration) and fault patterns are highly heterogeneous. The real source of complexity is the side‑effects of intelligence: more observability dimensions, finer configuration granularity, and higher policy‑update frequency, which overwhelm traditional manual‑heavy operations.

Defining System Intelligence

System intelligence means the data platform itself becomes an Agent that can perceive, diagnose, act, and evolve autonomously. It shifts the platform from passive command response to proactive intent understanding, embedding the Agent throughout the entire user‑platform lifecycle.

TCInsight: The Core Carrier of System Intelligence

TCInsight, Tencent Cloud's self‑developed data‑intelligence steward, implements full‑link system intelligence. It divides the platform lifecycle into four stages, each assisted by a dedicated Agent:

Resource Planning Agent (Selection Stage) : Replaces expert‑driven sizing with a dialogue where users provide data volume, task scale, and latency requirements; the Agent recommends cluster specs and cost combos automatically.

Intelligent Migration Agent (Cloud‑Migration Stage) : Works with automated migration tools to evaluate, translate SQL, move data, and verify regressions, shortening migration cycles by 50%.

Intelligent Control Agent (Daily Use Stage) : Enables natural‑language commands to query, configure, and manage the platform, eliminating menu navigation.

Autonomous Tuning Agents (Operations Stage) : Consist of three cooperating agents—autonomous tuning, autonomous operation, and predictive governance—that together achieve "auto‑driving".

Autonomous Tuning Agents in Detail

The three agents address the most labor‑intensive operations:

Autonomous Tuning Agent dialog‑drives Spark parameter optimization, turning years of expert trial‑and‑error into conversational adjustments and reducing resource waste by 15%.

Autonomous Operation Agent performs 7×24 health checks, auto‑diagnoses issues, and executes low‑risk fixes; mean time to root‑cause drops from 4.5 hours to 30 minutes.

Predictive Governance Agent monitors resource‑growth trends, issues pre‑emptive capacity alerts, and shifts incident handling from "post‑mortem" to "pre‑emptive".

All agents follow the ARRC principle—automate the repeatable, review the consequential—so high‑impact actions still require human approval and are fully rollback‑able with an emergency "dead‑hand" switch.

Knowledge‑Base Foundations

These agents rely on a continuously enriched knowledge base comprising a model library, scenario knowledge base, policy‑algorithm library, data‑feature store, and multi‑dimensional databases. Correct event labeling is enforced through confidence tiers, manual sampling, and periodic re‑validation to avoid label noise, model drift, and catastrophic forgetting. The approach has been recognized at VLDB 2025, and by 2025 TCInsight has handled over 100 000 system events across EMR, DLC, ES, TCHouse and third‑party platforms.

Typical Applications

Spark Task Auto‑Tuning : Detects OOM or disk‑space anomalies, pinpoints parameter bottlenecks, and suggests fixes via dialogue, with low risk and easy rollback.

Cluster Health Diagnosis : 24‑hour monitoring identifies health anomalies and narrows root‑cause time to minutes; human confirmation remains for remediation.

Storage Capacity Prediction : Forecasts storage growth, pre‑alerts capacity needs, and avoids production incidents, while managing false‑positive costs through configurable thresholds.

Conclusion

By embedding Agents at every stage—from selection and migration to daily use and operational tuning—system intelligence reduces dependence on scarce ops experts while keeping humans in the loop for consequential decisions, thereby enabling data platforms to manage themselves safely in the Agent era.