How Data Flywheels Accelerate Small Agentic Model Training

With the rapid advancement of large language models, agents are shifting from simple dialogue to task execution, yet face high inference cost, latency, and deployment barriers. To address this, the authors build an open‑source ecosystem based on the Qwen3 series, releasing multiple small agentic models, a synthetic RL dataset, and training code.

Data‑Environment‑Reward Co‑Design

They propose a collaborative "data‑environment‑reward" design for agentic RL, creating information gaps that force models to query, plan, and use tools, while replacing costly real APIs with stable mock environments.

A task‑level consistency mechanism ensures identical tool calls produce consistent responses within a task.

Mock Worlds: Synthetic Task Generation

Mock Worlds constructs low‑cost, scalable, verifiable synthetic tasks. By first generating a complete task goal and workflow, then rewriting it into an information‑deficient user instruction, the model must ask clarifying questions, invoke tools, and recover missing information, creating multi‑turn interactions and long‑chain planning challenges.

Environment Simulation

Each synthetic task is paired with a mock environment consisting of a mock user (provides hidden context) and a mock tool (simulates tool execution). This avoids the high cost and instability of real APIs and supports large‑scale RL rollouts.

Rubric‑Based Reward Construction

Rewards are derived from observable execution behaviors—completion of sub‑goals, required interactions, and avoidance of prohibited actions—rather than subjective scoring. Teacher model trajectories are aligned with task workflows to extract high‑level sub‑goals and construct task‑specific rubrics, providing stable, execution‑grounded supervision.

Data Flywheel Loop

The system iteratively improves data quality: failed samples are identified, augmented into harder variants (self‑instruction, persona injection, multi‑model consistency filtering), and fed back into training, continuously expanding the data distribution as the model improves.

Virtual Task Expansion via Behavior Trees

Four phases evolve tasks from linear flows to complex decision trees: (1) linear initialization, (2) behavior‑tree expansion with conditional branches, (3) back‑translation to generate new task instances from selected branches, and (4) adversarial mock users that inject misleading paths, forcing agents to verify facts before acting.

Experimental Evaluation

Across multiple training rounds, the data flywheel yields steady gains, especially for smaller models. Benchmarks such as TAU‑2 and BFCL‑V4‑Multi‑turn show improvements in tool usage, parameter handling, and long‑context robustness. AgenticQwen‑30B‑A3B demonstrates that small models can match or exceed larger bases when trained with this pipeline.

Resources

All models, datasets, and the Agentic trajectory pipeline are open‑sourced on ModelScope and HuggingFace, with usage examples via the ModelScope SDK and pip‑installable modelscope CLI.

References

Key related works include the ACL 2026 paper "Mock Worlds, Real Skills" and several EMNLP/ACL papers on model distillation and reasoning datasets.