How to Train a 671B‑Scale Model with RL: Insights from a verl Internship

1. Upstream‑downstream integration for RL training

Reinforcement‑learning (RL) training of ultra‑large models (e.g., 671 B parameters) requires a tight loop between a rollout inference engine and a training system that computes logs and gradients. This creates a dual‑dependency: the inference side must generate prompt‑response pairs, then the training side must ingest those pairs to update the model. Coordinating two distinct systems (inference + training) roughly doubles engineering effort and introduces integration complexity.

Typical training back‑ends for such scales are FSDP and Megatron (the verl stack does not yet support DeepSpeed). Inference engines commonly used are vLLM and SGLang . Selecting a framework impacts communication efficiency, GPU memory management, and parallelism strategy.

2. Framework feature support

When abundant resources (e.g., >1024 GPUs) are available, the DeepSeek paper or NVIDIA reference scripts can be followed directly. In realistic environments with far fewer GPUs, memory‑saving features must be integrated into the verl pipeline.

Training‑side enhancements

Initial support relied on a patched Megatron 0.4. NVIDIA assistance upgraded the stack to Megatron 0.11.

Non‑standard calls were removed and offload support for parameters, gradients, and optimizer states was added.

Megatron lacks native offload; custom offload logic was implemented.

Fragmentation was mitigated by exposing an environment variable to limit fragment size, e.g.:

export MEGATRON_MAX_FRAGMENT_MB=64

Inference‑side enhancements

Early vLLM versions loaded model parameters and KV cache together, causing OOM.

From vLLM 0.8.3 onward a staged wake‑up strategy loads parameters first and KV cache only when needed, reducing peak memory.

3. Model weight conversion pipeline

Weight conversion consists of two steps:

Checkpoint loading : FSDP can directly load HuggingFace checkpoints; Megatron requires its own checkpoint format.

Cross‑framework reshaping : After inference produces new data, weights must be reshaped to match the training framework’s naming and tensor layout.

Checkpoint formats

Legacy Megatron checkpoints bind the model to a specific parallelism strategy, making later re‑loading with a different strategy impossible. NVIDIA introduced a distributed checkpoint ( dist‑ckpt) format that decouples storage parallelism from training parallelism. Converting a 671 B HuggingFace checkpoint to a legacy Megatron checkpoint and then to dist‑ckpt typically requires ~256 GPUs and careful orchestration.

Tensor‑name mismatches

Megatron may fuse Q, K, V into a single tensor, while inference engines keep them separate. A reshaping (reshard) step maps keys from one convention to the other.

Parallel‑strategy mismatches

If training and inference use different parallel strategies, the conversion pipeline first gathers the full tensor size on each training GPU, then streams the appropriate slice to the inference side. This avoids extra alignment work and keeps tensor shapes intact.

Memory‑efficient conversion

Naïve conversion would allocate a full extra copy of the model, which is infeasible. Instead, a lazy‑load generator streams tensors one by one, ensuring peak memory usage never exceeds the size of a single tensor.

4. Precision alignment

To validate new features, developers run a trusted reference framework to obtain baseline outputs, then run the modified verl version and compare results. Small numerical differences can cascade into downstream failures, requiring extensive benchmark runs.

Challenges

Even sub‑percent deviations may break RL pipelines, leading to hours or days of debugging.

Many MoE configurations (e.g., DPSKV3) lack publicly available training‑ready checkpoints; only inference‑only checkpoints exist, which assert “not train,” making alignment difficult.

5. Model efficiency considerations

Running a 671 B model demonstrates feasibility but industrial adoption demands higher throughput (MFU) and lower training cost. Efficiency improvements include:

Expert‑parallelism (EP) for MoE layers.

Multi‑dimensional parallelism: tensor parallelism (TP), pipeline parallelism (PP), and expert parallelism (EP) combined.

Overlapping communication with computation.

Pipeline split‑and‑merge techniques.

Integrating these strategies coherently into verl remains an open research problem.

6. Practical integration issues

Because verl acts as a glue layer, bugs in upstream training or inference frameworks often surface as verl failures. Observed problems include:

SGLang’s GPU‑memory‑balance check fails under the RL + Ray setup, requiring direct coordination with SGLang developers.

vLLM’s MoE weight‑loading bug for certain models; patches must be applied and upstream maintainers notified.

Conclusion

Large‑scale RL training of a 671 B model highlights four critical engineering domains: (1) seamless coupling of inference and training pipelines, (2) memory‑efficient checkpoint handling and cross‑framework weight reshaping, (3) rigorous precision alignment against trusted baselines, and (4) systematic performance tuning via advanced parallelism. Continued collaboration with upstream projects and further engineering of verl are required to achieve production‑grade efficiency.