Agentic RL in Taobao Live: From RLVR to Multi‑Agent Reinforcement Learning

Taobao Live’s digital‑human interaction suffered from inflexible static workflow, weak context awareness, and high latency. To overcome these bottlenecks the system was rebuilt as an Agentic architecture that combines large‑language‑model world knowledge with autonomous planning, enabling global context perception, multi‑step tool calls, and reflective decision making.

Architecture Upgrade

Perception domain changed from single‑dimensional matching to full‑scene context (chat, product info, history).

Decision domain changed from one‑shot classification to iterative tool selection with self‑correction.

Execution domain changed from fixed intent scripts to multimodal responses (image, order, narration).

Latency dropped to 1.79 s (1.36 s reduction) and multi‑turn user ratio increased by 2.76 % .

Model Optimization: AgentTuning & RLVR

AgentTuning distills a 30B teacher’s planning, tool‑call, and reflection trajectories into a 30B‑A3B MoE student, cutting single tool‑call time to ~0.3 s and achieving 140 tokens/s throughput on a single H20 GPU.

RLVR introduces a domain‑specific metric and logical reward to suppress hallucinations, directly improving factual correctness and helpfulness of the reply model.

Multi‑Agent Reinforcement Learning

Because tool‑call ability and reply quality require different feedback, the authors split the system into two agents: a tool‑call agent and a reply agent. Each receives its own reward (tool‑call rationality vs. factual correctness & helpfulness) and is trained jointly via online collaborative RL.

Overall metric improvements: factual correctness + 4.1 pt , helpfulness + 23.6 pt , tool‑call rationality + 1 pt , total gain 8.2 pt .

Compared with the original workflow and top closed/open models (Gemini‑2.5‑pro, Qwen3‑235B‑A22B), the Agentic system shows superior performance in the live‑shopping scenario.

Training Pipeline (ROLL Framework)

Environment initialization: each rollout thread creates an isolated simulated live‑room, loads semantic vectors, and connects to remote tool services.

Rollout phase: the policy model interacts with the environment, issuing tool calls until a "get‑reply‑rule" signal terminates the tool‑call loop.

Reward evaluation: after rollout, an LLM judge scores factual correctness, helpfulness, tool‑call rationality, and length; group‑wise advantage is computed.

Model update: masked trajectories (only the generated tokens) are fed to a GRPO/PPO optimizer with KL‑regularization; updated parameters are synced back to the inference engine.

Masking removes teacher‑generated tool results and system prompts so that gradients affect only the policy‑generated tokens. Both agents use separate reference models for KL loss.

Experimental Results

Latency: Agentic (1.79 s) vs. Workflow (≈3.15 s); P99 tail latency also reduced.

Correctness & helpfulness: significant uplift over baseline and over Gemini‑2.5‑pro / Qwen3‑235B‑A22B.

Ablation: fixing the tool‑call agent and training only the reply model yields smaller gains (‑5.6 pt correctness, ‑6.6 pt helpfulness) compared to full Multi‑Agent RL.

Challenges & Future Directions

Simulation stability and reward discriminability are critical; noisy LLM judges can hinder convergence.

Current Multi‑Agent design follows a serial ReAct pattern; parallel swarm or sub‑agent architectures remain unexplored.

Training is task‑specific; future work aims at more general agents and skill‑based tool integration.

The authors conclude that the Agentic upgrade, combined with AgentTuning, RLVR, and Multi‑Agent RL, creates a high‑fidelity, highly interactive digital‑human live‑streaming paradigm and outline a roadmap for expanding toolsets and parallel multi‑agent capabilities.

// Example of a sanitized trajectory (JSON format)
[
  {"role": "system", "content": "## 指令
你是一名淘宝数字人主播..."},
  {"role": "user", "content": "## 真实输入
{\"当前商品名\": ...}"},
  {"role": "assistant", "content": "<tool_call>{\"name\": \"search_product_by_keyword\", \"arguments\": {\"keyword\": \"索尼xm4\"}}</tool_call>"},
  {"role": "tool", "content": "已找到相关商品：[...]"}
]