Introducing Vime-Ascend: An Open‑Source RL Framework with ModelArts Hands‑On Guide

Reinforcement‑learning (RL) frameworks that combine practical performance with open‑source availability are rare; slime offers a lightweight, efficient training design but lacks native vLLM support and NPU acceleration. Vime integrates slime’s training paradigm with vLLM’s inference engine, creating a unified pipeline, and the ascend branch—co‑developed with Huawei Cloud ModelArts—extends this pipeline to Ascend NPU hardware.

Architecture: three‑stage training‑inference decoupling

The pipeline consists of three cooperating modules:

Training (Megatron‑LM) : Executes the main training loop, reads data from the Data Buffer, updates parameters, and synchronises weights to the Rollout module. Megatron‑LM supplies comprehensive distributed optimisations (TP, PP, CP, EP, ETP) and flexible recomputation, ensuring efficient training of dense and MoE models.

Rollout (vLLM + vllm‑router) : Launches the vLLM inference engine, routes generation requests via vllm‑router, and produces samples with reward/verifier signals that are stored back into the Data Buffer. Optimisations such as PagedAttention, PrefixCaching, FP8 and PD‑separation are inherited directly from the vLLM community.

Data Buffer : Acts as a bridge, managing prompt initialisation, custom data handling and rollout generation methods, thereby decoupling training and inference and allowing independent scaling.

Core capabilities

High‑performance training : Full parallelism (TP/PP/CP/SP/EP/ETP) for dense and MoE models, dynamic batch packing (‑‑use‑dynamic‑batch‑size + ‑‑max‑tokens‑per‑gpu) that guarantees correct per‑sample loss, asynchronous training‑rollout pipelines, and automatic Megatron‑to‑vLLM weight synchronisation.

Flexible data generation : DynamicSampling with oversampling and filtering (e.g., check_reward_nonzero_std), Partial Rollout for caching aborted generations, multi‑turn interaction and tool‑calling via custom generate and reward functions, and PD‑separation deployment (‑‑prefill‑num‑servers).

Algorithm and model coverage : Built‑in GRPO, GSPO, Reinforce++ (and baseline), PPO, and support for Qwen series (3.6, 3.5, 3Next, 3MoE, 3, 2.5), DeepSeek‑V3, Llama‑3, GLM‑4.5‑Air, among others.

Multi‑hardware abstraction : Unified resource abstraction enables the same RL pipeline to run on GPU or NPU without code changes.

Ascend integration

The ascend branch provides:

Training backend: Megatron‑LM + MindSpeed + Megatron‑Bridge with TP/PP/CP/EP and dynamic batch.

Inference backend: vLLM + vllm‑ascend with HCCL native weight synchronisation.

Weight conversion: default ‑‑megatron‑to‑hf‑mode bridge exports checkpoints directly consumable by vLLM, preserving log‑prob alignment.

Co‑locate mode: 8‑card training + vLLM rollout on a single node, using Torch‑Memory‑Saver and vLLM CaMem sleep/wake for memory sharing; validated on A2 with a 500‑step long‑run.

Container support: Dockerfile.npu and pre‑built images (quay.io/ascend/vime:vime-a2-latest for A2, vime‑latest for A3) with CANN 9.0.0, vLLM‑Ascend and MindSpeed.

Experimental validation on Ascend NPU

Training‑inference separation (A2) – Qwen3‑4B with GRPO, global‑batch‑size 256, rollout‑batch‑size 8, 4‑card training + 4‑card rollout:

train_rollout_logprob_abs_diff stayed between 0.010 – 0.015 (mean ≈ 0.012), indicating stable log‑prob alignment.

raw_reward rose from ~‑1.0 early on, while the truncated‑ratio decreased, showing a clear upward reward trend.

Overall end‑to‑end step time: 314 s for Vime vs ~1000 s for the baseline (≈ 2.18× speed‑up) with consistent per‑step timing.

Co‑locate mode (A2) – 8‑card unified training + rollout on Ascend 910B1:

train_rollout_logprob_abs_diff started at ~0.012, averaged ~0.008 over 500 steps, staying within 0.006 – 0.0125 and converging to ~0.007.

No OOM or training interruptions during the 500‑step run; reward and entropy metrics evolved normally, confirming effective learning signals.

Why the NPU design matters

Decoupled training and inference reduce adaptation complexity; each side can be optimised for NPU independently.

vLLM’s multi‑hardware ecosystem brings mature NPU inference to Vime without building a stack from scratch.

Unified abstraction allows a single codebase to run on GPU or NPU, with the ascend branch maintaining parity with the main line.

vllm‑router’s load‑balancing works naturally on multi‑card NPU setups via the ‑‑router‑policy flag.

Quick start

Pull a ready‑made Docker image:

export IMAGE=quay.io/ascend/vime:vime-a2-latest   # A2
# export IMAGE=quay.io/ascend/vime:vime-latest   # A3
docker pull "${IMAGE}"

Or build from source:

git clone -b ascend https://github.com/vllm-project/vime.git
cd vime
docker build -f docker/Dockerfile.npu -t vime-ascend:a2 .

Core parameters are grouped into three categories:

Megatron parameters (e.g., --tensor-model-parallel-size 2).

vLLM parameters prefixed with --vllm- (e.g., --vllm-gpu-memory-utilization 0.7).

Framework parameters (e.g., --rollout-batch-size 16, --advantage-estimator grpo).

Run the Qwen3‑4B GRPO example:

bash scripts/run-qwen3-4B.sh          # GPU reference
bash scripts/models/qwen3-4B_npu.sh  # NPU (A2/A3)

Future directions

Support for next‑gen Ascend A5 (950) hardware.

Integration of Transfer Queue to replace the Data Buffer.

Continued MoE routing‑replay validation for Qwen3‑30B/32B on NPU.

Implementation of PD‑separation across NPU nodes.

Further performance tuning of weight synchronisation, router load‑balancing and dynamic batching for NPU.

ModelArts end‑to‑end workflow

1. Build the custom image (Dockerfile.npu) and push it to Huawei Cloud SWR.

2. Upload model checkpoints, training data (e.g., dapo‑math‑17k) and scripts to OBS.

3. In ModelArts console, create a training job, select the custom image, set the start command to run the provided NPU script (e.g., run_npu_qwen3_4b.sh), and configure environment variables such as RAY_EXPERIMENTAL_NOSET_ASCEND_RT_VISIBLE_DEVICES.

4. Allocate resources (Ascend A2/A3, 4‑card training + 4‑card rollout or 8‑card co‑locate) and launch the job.

5. Monitor logs, checkpoints and metric curves directly in the console; the same configuration reproduces the results reported in sections 1.6 and 1.7.

Conclusion

Vime‑Ascend delivers a concise, stable and efficient RL pipeline for large models on Ascend NPU, achieving up to 2.18× speed‑up over a baseline while maintaining log‑prob alignment. The article walks readers through the framework’s architecture, benchmark results, practical Docker‑based deployment, and a complete ModelArts cloud‑native workflow, enabling developers to reproduce and extend the presented experiments.