AReaL v1.0.5: Colocated Training/Inference & Multi-Teacher On-Policy Distillation for Efficient RL

AReaL-Ascend v1.0.5 adds two capabilities that address resource efficiency and capability fusion for reinforcement learning on Ascend NPUs.

Colocated Training & Rollout (训推共卡)

Traditional RL separates Trainer and Rollout Engine onto independent NPU pools. The new colocated mode lets training and vLLM rollout reuse the same NPUs by alternating phases, lowering the resource barrier for RL experiments.

This mode suits three scenarios:

NPU-constrained environments: no separate rollout pool required.

On-policy training: the latest policy rolls out immediately after training, reducing policy staleness.

Fast debugging and reproducible baselines: simpler resource topology.

AReaL now supports both deployment styles: disaggregated for large-scale asynchronous throughput, and colocated for resource efficiency, strict on-policy, and quick reproduction.

Implementation: Ray Scheduling + AWEX IPC Zero-Copy

The core flow:

Ray NPU Bind → Training → Weight Offload → Stride Fork → Launch vLLM on Same NPUs → Rollout → AWEX Weight Update

Key optimizations:

NPU Bind: Ray manages NPUs; each training worker binds one NPU, establishing stable device mapping for later inference reuse.

Stride Fork: reorganizes a group of training workers into a vLLM inference instance matching the desired tensor-parallel size, reusing the original Ray placement.

IPC Zero-Copy: inference processes directly map training-process NPU weight memory via IPC handles, avoiding NPU→CPU→NPU data movement.

Group Tensor Packing: merges tensors by shape/dtype before IPC serialization, drastically cutting IPC handle count and system overhead for large models.

Recursive Partition: reshards training weights per the AWEX transfer plan, sending only required shards to each inference rank, supporting mismatched training/inference parallel strategies.

The zero-copy path saves tens of GB of transfer and seconds of latency for a 27B model.

Runnable Example: Qwen3.6-27B Single-Node Colocated

Config:

https://github.com/HwVanICI/AReaL/blob/ascend-v1.0.5/examples/vlm_npu/qwen3_6_27b_geometry3k_grpo_awex_colocate.yaml

python examples/vlm/geometry3k_grpo.py --config examples/vlm_npu/qwen3_6_27b_geometry3k_grpo_awex_colocate.yaml

The example runs Qwen3.6-27B + Geometry3K + GRPO + AWEX Colocate on a single node, achieving stable 200-step multimodal RL training.

Multi-Teacher On-Policy Distillation (MOPD)

Beyond agentic RL, on-policy distillation is becoming a key post-training paradigm. The challenge: how to merge multiple domain-expert models (e.g., math, code, search) into one generalist student.

Traditional offline distillation suffers from train-test distribution mismatch. MOPD solves this by letting the student generate on-policy, then having each domain teacher score the student's actual tokens in real time.

MOPD Core Loop

Train independent expert/teacher models per domain.

Do not use teacher-generated offline data.

Student performs on-policy rollout.

For tokens the student actually generates, the corresponding domain teacher provides token-level scores.

Convert teacher-student log-probability gaps into dense optimization signals.

Jointly optimize RL objective and distillation objective.

Advantage: teachers score only samples from the student's current policy, reducing both train-test and teacher-student distribution mismatch.

Plain-Language Analogy

Offline distillation: teacher prepares an answer key; student practices only the key's path. If the student deviates, no reference exists.

MOPD: student solves live; whatever step the student writes, the teacher grades that exact step on the spot.

MOPD has shown effectiveness in multi-domain integration and was used in MiMo-V2-Flash industrial post-training.

AReaL-Ascend MOPD Example

Docs:

https://github.com/HwVanICI/AReaL/blob/ascend-v1.0.5/examples/distillation/mopd/README.md

Current example uses two domains:

AIME: math reasoning.

LeetCode: code generation.

Validated on 2× Atlas 800T A3 nodes (16 NPUs). Results after 25 steps:

AIME Reward: 0.51 → 0.75

LeetCode Reward: 0.31 → 0.69

Domain-Aware Teacher Routing

AReaL routes trajectories by dataset domain to the matching teacher:

AIME trajectory → AIME Teacher
LeetCode trajectory → LeetCode Teacher

Each expert focuses on its specialty, improving signal relevance. Additionally, mixture routing allows multiple teachers to score the same trajectory, combining their token log-probabilities via weighted fusion for more flexible, powerful distillation.

RL + Distillation Joint Optimization

MOPD is not pure imitation. Student rolls out first; teachers evaluate generated tokens. Per-token teacher signal:

Teacher favors token → increase token's learning advantage
Teacher disfavors token vs. student → decrease token's learning advantage

This token-level feedback provides denser, timelier guidance than sparse terminal rewards.

In AReaL, the actor objective combines: RL Loss + MOPD Distillation Loss Thus the model leverages:

Reward signal: whether the final task succeeded.

Teacher token-level signal: which tokens align with domain-expert policy during generation.

Together they keep the end goal on track while mitigating sparse, unstable pure-RL signals, improving convergence speed and quality.

This enables scalable expert-fusion pipelines: separate teams train Math, Code, Search, Agent experts, then MOPD incrementally integrates them into one generalist student — a practical, engineering-ready path for "expert capability fusion."

Huawei Cloud ModelArts now natively integrates the AReaL framework, offering one-click access to these capabilities, with future AReaL features to be exposed as cloud services.