ActionOPD: Distilling Agent Decisions, Not Tokens — Tsinghua's Action-Level Online Distillation

In multi-turn tool use and complex environment interactions, large language models are increasingly deployed as agents. Constrained by the high inference cost of frontier models, transferring their decision-making ability to lightweight models has become a key research topic. Unlike single-turn text generation, each agent step changes the environment state and affects subsequent trajectories, raising two fundamental questions: (1) In what state to supervise — on the teacher's preset trajectory or on states the student actually encounters? (2) At what granularity to align — token-level vocabulary distribution matching or high-level semantic action alignment?

Limitations of Existing Paradigms

The article reviews two mainstream agent distillation paradigms:

Offline Knowledge Distillation (KD/SFT) : Imitates teacher tokens on teacher trajectories. Because training does not cover mistake states, student performance drops sharply when it deviates from the preset trajectory due to state distribution shift.

Token-level Online Policy Distillation (Token-level OPD) : Aligns teacher token distributions on student-sampled trajectories. While it mitigates state shift, it strongly depends on teacher white-box log-probs and a shared tokenizer, making it unsuitable for black-box or cross-family distillation, and still forces token-by-token imitation of teacher reasoning.

Both methods supervise primarily at the token level. ActionOPD's core insight: in multi-turn interaction, what truly drives environment state transitions are semantic actions , not specific token sequences. An agent's single-step output contains internal reasoning (which does not directly act on the environment) and external actions — tool calls (determined by function and parameter semantics) and user replies (determined by communicative intent and key facts). Both admit multiple equivalent expressions. For example, "Basic economy class cannot be changed" and "This fare type does not support changes" differ in wording but express the same restriction. Therefore, to align the student at the decision level rather than surface form, the distillation objective should not be fitting a specific token sequence but maximizing the student's overall probability of taking a teacher-synonymous action in the current state .

ActionOPD Method

2.1 Deriving Policy Gradient from Action Likelihood

Since the same semantic action can be realized by multiple reasoning paths and phrasings, the student's total probability of taking a given semantic action equals the sum of probabilities of all reply sequences expressing that semantics. Although direct enumeration is infeasible, the paper proves a concise equivalence: in the states actually visited by the student, taking the gradient of the expected teacher action likelihood converts exactly into the classic REINFORCE policy gradient — where "whether the student's reply is semantically consistent with the teacher's action" serves as the reward signal, efficiently estimable via finite sampling.

This property yields two direct advantages:

Pure black-box interface and cross-family compatibility : Only requires sampling replies from teacher and student, judging whether their actions are semantically equivalent to construct the distillation reward — no teacher token log-probs or shared tokenizer needed .

Decoupling action supervision from autonomous reasoning : The action-level reward uniformly weights all tokens generated in a single step, directly reinforcing the student's self-generated reasoning chains and action expressions , without requiring token-by-token imitation of the teacher's chain of thought.

2.2 State-Group Rollout and Policy Update

To ground the theory in multi-turn training, ActionOPD designs a State-Group Rollout mechanism:

Parallel sampling of teacher and student actions per state : At each visited state, sample multiple teacher actions and a group of student candidate replies in parallel.

Action-level semantic matching and scoring : Ignore internal chain-of-thought; extract only external actions for semantic comparison — tool calls via parameter normalization to discriminate operational semantics; user replies via LLM comparison of communicative intent and key facts. Matching a teacher sample yields a base positive reward; matching multiple teacher samples gives a small frequency bonus; illegal outputs receive negative penalty.

Environment advancement and policy update separation : When advancing the environment, submit only the highest-scoring action in the group, using teacher preference to guide student exploration of more informative states; when updating parameters, reuse all candidates in the group to compute normalized relative advantages and update student weights via a PPO-style clipped objective.

Experimental Results

The team validates the "learn decisions, not tokens" paradigm across three dimensions: cross-benchmark and cross-family generalization, same-family controlled comparison against offline KD and token-level OPD, and real interaction trajectory analysis.

Cross-Benchmark Zero-Shot Transfer and Cross-Family Scaling

On AWM and EnvScaler, using same-family Qwen3.7-Flash and heterogeneous DeepSeek-V4-Flash as teachers to distill Qwen3-4B/8B students. As shown in Figure 3 and Table 1, all four configurations achieve large gains over base models on four unseen benchmarks in zero-shot generalization, exhibiting clear bidirectional scaling laws: (1) 8B students consistently outperform 4B; (2) the stronger heterogeneous teacher DeepSeek-V4-Flash yields higher performance on both student sizes than the same-family teacher, proving ActionOPD breaks vocabulary constraints and lets teacher selection focus on decision quality rather than architectural homology.

Comparison with Offline KD and Token-level OPD

On τ²-bench in a same-family controlled experiment (Qwen3-32B → Qwen3-4B), Table 2 shows ActionOPD achieves performance comparable to KD/OPD baselines without requiring teacher token probabilities : test-set single-run success rate (pass¹) improves 10.8 percentage points over base, leading both baselines; four-run full success rate (pass⁴) nearly doubles base, though slightly below the token-constrained baselines, trading a small margin for the flexibility of black-box text-only interface and native cross-family support.

Behavioral Improvements in Real Interaction Trajectories

Pre/post training trajectory comparisons reveal three typical behavioral shifts confirming decision-policy optimization:

Adhering to business rule constraints : In aviation scenarios facing non-changeable basic economy fares, base model blindly calls API to illegally rebook; ActionOPD strictly follows rules and compliantly transfers to human agent.

Accurately distinguishing information source from operation target : In retail scenarios requiring address extraction from historical order to modify current order, base confuses source and target orders causing missed updates; ActionOPD correctly executes "query source address → modify target order" chain.

Completing operation loops and result verification : In telecom scenarios where payment does not auto-reactivate service, base only verbally promises restoration but omits tool call; ActionOPD proactively calls resume_line to reactivate and guides user to verify SIM card.

Self-ActionOPD: Self-Distillation via Privileged Context

Since ActionOPD only requires aligning external action semantics without imitating internal reasoning, a natural question arises: Can a single model self-distill by leveraging an "information gap" without a stronger external teacher?

In Self-ActionOPD , the same Qwen3-4B serves as both teacher and student: the teacher receives additional privileged context — undisclosed user goals, implicit rules, and reference action prompts — while the student receives only public context. Because alignment targets only external actions, the student need not mimic the teacher's privileged-information-based internal reasoning; it directly learns to make decisions consistent with the "open-book teacher" under public context. After training, privileged information is removed and the model is tested independently on public context only.

Table 3 shows self-distilled Qwen3-4B improves test-set pass¹ by 4.8 percentage points and pass⁴ by 2.5 percentage points , with gains across all three sub-domains, demonstrating that the model can internalize high-quality decisions from a "privileged perspective" into default capability, providing a viable path for agent self-evolution.

Conclusion and Outlook

Centered on the "learn decisions, not tokens" principle, ActionOPD transforms expected teacher action likelihood into online policy optimization with semantic rewards, enabling cross-family online distillation and same-model self-distillation using only black-box text interfaces. Future directions include combining environment verification signals to filter teacher errors, reducing complex action matching errors, and improving sampling coverage in states with multiple valid paths. Code and model weights are open-sourced at the project page and GitHub repository.