How Tencent Ads Achieved Full‑Automation CI/CD with AI‑Driven Monitoring

Background – Why Automate CI/CD

Tencent Ads' advertising technology team faced high labor costs, low release quality, slow iteration speed, and limited efficiency data, prompting a shift from human‑driven to data‑and‑algorithm‑driven CI/CD.

Problems Addressed

Human Cost : Repetitive manual release tasks consumed significant manpower.

Release Quality : Thousands of services with complex dependencies required manual monitoring, leading to errors.

Iteration Speed : Manual interventions capped the release frequency.

Efficiency Measurement : Lack of a global quality funnel prevented data‑driven improvements.

Unattended CI/CD Before and After

Before: a skeletal CI/CD process with many manual steps and inaccurate release blocking.

After: zero‑human release operations, full coverage of all modules, standardized deployment, and automated gray‑scale releases.

Core Solutions

Full‑Process Automation : Standardized pipelines (compile → test → deploy) integrated with TAPD, GIT, and BlueShield CI, enabling end‑to‑end automation.

AI Interception : Large‑model AI analyzes change metrics, automatically blocks risky releases, and notifies owners.

Transparent Dashboards : Real‑time visual boards show MR, pre‑merge checks, integration tests, and deployment status.

Metric Classification : Module, platform, modulation, and basic metrics are defined, with smart thresholds learned from historical data.

AI Monitoring Strategy

AI models automatically set thresholds, perform multi‑dimensional anomaly detection, and reference a knowledge base of past incidents to reduce false positives.

Implementation Framework

Data Integration Server : Collects real‑time metrics.

Scheduling Server : Triggers anomaly detection agents.

Anomaly Detection Agent : Identifies metric deviations without manual thresholds.

Anomaly Aggregation Agent : Uses RAG and function calls to reason about single and multiple metric anomalies.

Knowledge Base : Stores operational experience for AI reference.

Results

Automation reduced manual release effort by over 80%, increased test automation coverage to 63%, cut release time from half an hour–two hours to near zero, and lowered missed interceptions by 85%.

Key Metrics

Leakage Rate : (Rollback count – invalid rollbacks) / release count.

Interception Rate : (Failed releases – non‑effective interceptions) / release count.

False‑Positive Rate : (Successful releases with alerts) / release count.

Future Directions

Continue refining AI thresholds, expanding metric coverage, and improving multi‑metric correlation to further enhance release safety and speed.