Skip to content
Tagged articles

Megatron-LM

7 articles · Page 1 of 1
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Nov 1, 2026 · Artificial Intelligence

LoRA on Decoder Only: Half Parameters Match Full Fine-Tuning on DeepSeek-V4.1-Flash

The article explores optimal LoRA placement for DeepSeek-V4.1-Flash's causal encoder-decoder architecture, finding that applying LoRA only to decoder layers (20–39) with rank 16 matches full-model performance (0.296 vs 0.293 validation loss) while halving adapter parameters and enabling shared KV cache across adapters by freezing layer 20's KV projection.

DeepSeek V4.1 FlashKV cache sharingLoRA
0 likes · 32 min read
LoRA on Decoder Only: Half Parameters Match Full Fine-Tuning on DeepSeek-V4.1-Flash
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
Jun 26, 2026 · Artificial Intelligence

Introducing Vime-Ascend: An Open‑Source RL Framework with ModelArts Hands‑On Guide

This article presents Vime-Ascend, an open‑source reinforcement‑learning framework that merges slime's training design with vLLM's inference engine, details its three‑stage decoupled architecture, showcases NPU performance gains on Qwen3‑4B GRPO experiments, and provides step‑by‑step ModelArts deployment instructions.

AscendMegatron-LMModelArts
0 likes · 22 min read
Introducing Vime-Ascend: An Open‑Source RL Framework with ModelArts Hands‑On Guide
360 Smart Cloud
360 Smart Cloud
Jul 17, 2024 · Artificial Intelligence

Parallelism and Memory‑Optimization Techniques for Distributed Large‑Scale Transformer Training

This article reviews the principles and practical implementations of data, pipeline, tensor, sequence, and context parallelism together with memory‑saving strategies such as recomputation and ZeRO, and demonstrates how the QLM framework leverages these techniques to accelerate large‑model training and fine‑tuning on multi‑GPU clusters.

GPUMegatron-LMParallelism
0 likes · 18 min read
Parallelism and Memory‑Optimization Techniques for Distributed Large‑Scale Transformer Training
360 Smart Cloud
360 Smart Cloud
Jan 26, 2024 · Artificial Intelligence

Parallel Strategies for Distributed Deep Learning Training

This article reviews distributed training techniques for large deep‑learning models, covering data parallelism, model parallelism (including pipeline and tensor parallelism), gradient bucketing and accumulation, 3D parallelism, and practical implementations such as Megatron‑LM and 360AI platform optimizations.

AIMegatron-LMdata parallelism
0 likes · 22 min read
Parallel Strategies for Distributed Deep Learning Training
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Dec 5, 2023 · Artificial Intelligence

How to Efficiently Fine‑Tune Qwen LLMs on Alibaba Cloud PAI Lingjun

This guide walks you through setting up Alibaba Cloud PAI Lingjun resources, preparing Qwen‑7B/14B/72B models, preprocessing large‑scale WuDao data, configuring distributed training with Megatron‑LM, performing continued pre‑training and supervised fine‑tuning, and finally deploying the model as an online service via PAI‑EAS.

Alibaba CloudLLMMegatron-LM
0 likes · 27 min read
How to Efficiently Fine‑Tune Qwen LLMs on Alibaba Cloud PAI Lingjun
Alimama Tech
Alimama Tech
Sep 12, 2023 · Artificial Intelligence

Megatron-LLaMA: High-Performance Large Language Model Training Framework

Megatron-LLaMA is an open‑source high‑performance training framework for LLaMA models, offering tensor, pipeline, and sequence parallelism, an overlapped optimizer, and near‑linear scalability, achieving up to 176% speedup on 32 GPUs and robust performance even with limited network bandwidth.

DeepSpeedGPU optimizationLlama
0 likes · 10 min read
Megatron-LLaMA: High-Performance Large Language Model Training Framework
Architects' Tech Alliance
Architects' Tech Alliance
Aug 31, 2022 · Artificial Intelligence

Performance Evaluation of Transformer Models on the Inspur NF5488A5 GPU Server

This article presents a detailed benchmark of four Transformer models of varying sizes trained on the high‑end Inspur NF5488A5 GPU server, compares its NVSwitch‑based interconnect with a PCIe‑based system, and analyzes the impact of model scale, tensor parallelism, and hardware bandwidth on training efficiency.

DeepSpeedGPU serverMegatron-LM
0 likes · 12 min read
Performance Evaluation of Transformer Models on the Inspur NF5488A5 GPU Server