Inside iQIYI's Agent Team: Automating Cross-Service Big Data Diagnosis

iQIYI's big data ecosystem spans over 20 services and platforms, generating heavy operational load from user queries like "Why did this Spark task fail?" or "Why is StarRocks query slow?" Manual triage requires reading chat context, querying platform data, searching logs and alerts, analyzing metrics, and correlating historical experience — often across compute, storage, scheduling, network, and downstream systems.

Initially, the team codified common troubleshooting flows into rule-based automation workflows. However, real incidents involve incomplete information, multiple services, and dynamic investigation paths that preset rules cannot cover. AI Agents suit this: they analyze clues around a goal, autonomously select and combine tools, and adjust reasoning based on execution results.

Agent Team Architecture

The Big Data Assistant adopts a Coordinator + multiple Service Agents model:

Dialoguer : Handles user communication, understands intent, organizes final reply.

Coordinator : Team scheduler; based on user goal and team blackboard, decides next action — ask follow-up, dispatch task, or finish.

Service Agents : Domain experts (Scheduling Platform, Batch Computing/Spark, Real-time Computing/Flink, Machine Learning, Data Lake, Data Analysis/StarRocks). Each maintains its own skills, knowledge base, and workspace; they only process assigned tasks and return structured results.

Agents share a Team Blackboard — a structured collaboration space where Coordinator writes tasks and context, Service Agents update progress and evidence, and Dialoguer reads converged information. This separates shared facts from each agent's private execution history, avoiding context pollution.

Cross-Service Collaboration Example

A Flink job writing to StarRocks fails frequently. The flow:

Coordinator assigns to Flink Agent.

Flink Agent finds exceptions concentrated in downstream write phase, extracts timeout evidence.

Coordinator hands off to StarRocks Agent with time window and node clues.

StarRocks Agent aligns transaction and resource state, confirms write request backlog and slow node response.

Coordinator merges evidence, explains root cause, impact scope, and next steps.

For independent sub-tasks (e.g., Iceberg table data latency), Coordinator dispatches to Flink Agent (check job logs, checkpoints) and Iceberg Agent (inspect table files, metadata) in parallel. They execute in isolated workspaces, continuously writing clues to the blackboard — enabling "execute, discuss, calibrate" rather than waiting for all to finish.

Architecture Evolution

Phase 1: AI-Enhanced Workflow

Early stage used workflows with LLM nodes for intelligent inspection. Deployed for 1,500+ key compute tasks and 90+ critical pipelines (content popularity, ad effectiveness, live commerce, CDN stats). Workflows handled single-service Q&A and hourly anomaly patrols, reducing alert fatigue. But multi-service joint diagnosis, multi-dimensional log/metric analysis, and multi-turn follow-ups caused workflow node count to exceed 50, becoming inflexible and hard to maintain.

Phase 2: Single Agent on Feishu Aily (MCP)

Moved to Feishu's low-code agent platform Aily. Main loop: receive dialogue → reasoning loop (select tools based on intermediate results) → compose reply. Service capabilities integrated via MCP (Model Context Protocol) for standardized tool/data connection. Served single-service diagnosis and Q&A for months. Bottlenecks: tool calling and context management were black boxes, hard to tune; graphical development hindered code reuse, testing, version control, and automation.

Phase 3: Skill-Based Agent Team on LangGraph

Adopted Skill pattern: Model + Harness builds core agent flow; Skills implement business capabilities. Switched runtime from Aily to LangGraph, upgraded from single Agent to Agent Team. Services expanded from one to many. One Agent Team process lets different service capabilities extend independently by responsibility and collaborate under a unified team protocol.

Layered Architecture

Entry Layer : Receives requests from CLI, Web, IM; handles protocol adaptation and presentation only.

Scheduling Layer : Coordinator understands overall goal, decomposes tasks, dispatches to Service Agents.

Runtime Layer : Hosts Service Agents' multi-step execution flows and state, enabling continuous tool use.

Capability Layer : Domain Skills and shared Tools (execute commands, call service APIs, query knowledge bases).

Harness Layer : Governs runtime context, file I/O, tool invocation control, execution budget, exception handling and retry.

Harness Engineering Details

1. Fact Source Layering

Facts separated into three tiers: external real dialogue, internal team tasks/evidence, and per-agent private workspace. Roles read only what they need, preventing internal execution details from polluting user intent and enabling separate context governance.

2. Read-More-Write-Less on Shared Facts

Agents never freely rewrite shared context. Each model call assembles a temporary view from real dialogue, team blackboard, and workspace — used only for that call, not persisted as long-term fact.

3. Cold/Hot Context Governance

Continuous troubleshooting may span 100+ tool calls, bloating context. Current execution = hot context (kept in Service Agent); older history = cold context (offloaded to filesystem with on-demand query). Preserves continuity without diluting model attention; retains raw evidence better than threshold-based compression.

4. Strict Tool Output Control

Tool outputs dominate context usage. Framework intercepts results: if over threshold, full result saved to local file; model receives summary preview, file location, and metadata. Model reads relevant segments on demand. Limits per-call impact while preserving complete evidence.

5. Tool Lazy Loading

As tool count grows, descriptions consume context. Low-frequency tools expose only name and brief description; full usage and parameters loaded only when model confirms need. Keeps regular paths focused. Still, tool count must be restrained — each added tool can reduce calling accuracy and increase token cost. Team experienced adding then reducing tools.

6. Structured Contracts for Reliable API Calls

Services expose many APIs; models may misconstruct URLs, omit parameters, or waste context reading docs. Solution: declare allowed APIs in catalog with unique IDs; Skills reference only needed operation IDs; Agent fetches parameter specs on demand; model generates business parameters; framework assembles, validates, and calls service. Skills focus on business steps; model only picks operation and fills business params; framework handles request mechanics. Reduces context cost and improves success rate.

7. Execution Progress Control

Agents can over-invest in local directions, appearing "stuck". Two mechanisms: periodic reporting (every N steps, summarize progress and remaining tasks to re-check direction) and model call budget (warn near threshold, force stop at threshold as safety net).

8. Langfuse Trace Observability & Quality Analysis

Traditional logs cannot reconstruct why an agent made a judgment. Langfuse records full trace from user question to final reply, linking role dispatch, model calls, tool executions, errors, latency, and results. Quality evaluated on correctness, usefulness, adherence; aggregated into dashboards for trend monitoring and anomaly detection. Low-quality traces can be inspected to pinpoint failure stage (task understanding, service routing, tool calling, conclusion convergence) and verify optimizations.

Deployment Forms

Two interaction modes sharing same Agent Team and domain capabilities:

Feishu DM : Continuous dialogue with recent history auto-loaded, older messages forgotten per policy; supports topic switching.

Feishu Group @BigDataAssistant : Reads group history for context, replies when mentioned; continuous dialogue requires repeated @.

Web AI Floating Ball : On big data platform pages, click floating ball to chat without leaving; first message auto-carries current page context.

Summary & Roadmap

The assistant has become an indispensable intelligent partner, shifting support from "find people, find docs, find platforms" to "describe problem, assistant collaborates," lowering user support cost and reducing service owner ops burden.

Current limitations: quality evaluation loop (identify → categorize → optimize → verify) needs closure; permission/risk controls prevent direct cluster login for final verification, still requiring service owners; user-side task management basic — scheduled triggers, multi-issue management, in-execution interaction guidance need improvement.

Future goals: expand service coverage to more big data platforms/domains; close quality optimization loop for continuous discovery, attribution, correction, and regression validation; enhance task management and process interaction with scheduling, multi-issue handling, and timely guidance.