AReaL v1.0.5 LoRA RL: Low-Rank Adaptation for Accessible Large Model RL on Ascend

Introduction

This article concludes the AReaL-Ascend v1.0.5 series by introducing LoRA-based Reinforcement Learning, a key capability that lowers the resource barrier for large model RL post-training, and provides a quick-start guide with documentation links.

Why LoRA Matters for RL

LoRA's core value is training only a small number of low-rank adapter parameters (~1% of total) instead of updating the full base model. For a 27B parameter model, full-parameter training requires maintaining complete gradients, optimizer states, and activations, incurring massive GPU/NPU memory overhead. LoRA retains most performance gains while dramatically reducing memory pressure.

This is especially critical for RL because RL resource overhead comes not only from training but also from:

Rollout inference

Actor / Reference model

Optimizer states

Gradient / Activation

Policy weight synchronization

Therefore, compared to full-parameter RL, LoRA significantly reduces training-side memory pressure and the amount of trainable parameters that must be saved and transferred, enabling larger models to run RL post-training on fewer hardware resources.

LoRA Documentation

Reference documentation:

https://github.com/HwVanICI/AReaL/blob/ascend-v1.0.5/docs/en/reference/lora.md

Megatron LoRA Weight Update Modes

AReaL's Megatron LoRA supports two rollout weight update modes.

Mode 1: Training and Rollout Both Use LoRA Adapter

Base Model  + LoRA Adapter trained by Megatron  ↓ LoRA Adapter sync  ↓ vLLM Base Model + LoRA Adapter

Megatron trains the LoRA adapter; vLLM dynamically loads and applies the adapter during rollout. This mode currently targets dense models.

Mode 2: LoRA Merge Then Sync to vLLM

Base Model + LoRA Adapter  ↓ Merge  ↓ Merged Model Weights  ↓ vLLM

Megatron still trains only LoRA parameters, but at policy update the adapter is merged back into the base weights, and the merged weights are provided to the rollout engine. Advantages: vLLM rollout side does not need to run LoRA adapters directly, and this mode supports MoE (Mixture of Experts) scenarios.

Additional Features

Cross-node LoRA RL with Megatron + vLLM

LoRA fine-tuning for MoE models such as Qwen3 MoE

XCCL-based LoRA weight communication

Qwen3.6-27B single-node and multi-node LoRA RL examples

Series Summary

The AReaL-Ascend v1.0.5 series covered three core capabilities:

Qwen3.6 Code Agent 64K RL + THD/MTP optimization — enabling long-sequence agent training

Training-inference co-location + MOPD — dual breakthrough in resource efficiency and capability fusion

LoRA RL + quick start — lowering the barrier to make RL accessible

AReaL-Ascend v1.0.5 marks an important step from "lab technology" to "engineering deployment" for large model RL post-training.