ChunkFlow: Accelerating Long‑Context Model Fine‑Tuning Up to 4.5× Faster

Recently, the Alibaba Cloud PAI team, Tongyi Lab, and UCAS Frontier Interdisciplinary Science Institute co‑authored a paper titled "Efficient Long Context Fine‑tuning with Chunk Flow" presented at ICML 2025.

ChunkFlow is a solution for efficient training on variable‑length and ultra‑long sequence datasets, supporting the Qwen series of large language models. It delivers more than 2× end‑to‑end performance gain in internal Alibaba Cloud workloads and up to 4.53× speedup compared with other frameworks.

Research Background

Long‑text capability is a core ability of language models and is essential for many downstream tasks. Continue pre‑training and long‑context fine‑tuning are key to extending this ability. Real‑world datasets exhibit a heavy‑tailed length distribution, with most samples short and a few extremely long, causing GPU under‑utilisation and pipeline bubbles.

Existing Problems

Fixed memory and parallelism strategies conflict with varying sequence lengths, leading to load imbalance and performance degradation.

Ultra‑long sequences create huge activation memory pressure, requiring recomputation or off‑loading and further reducing pipeline efficiency.

Paper Contributions

ChunkFlow reorganises data into fixed‑size chunks. Short sequences are concatenated, long sequences are split, and a state‑aware scheduler guarantees correct attention computation while controlling memory usage.

Algorithm 1 (illustrated below) shows how training chunks are built.

For split long sequences, causal attention requires careful ordering; the scheduler (Algorithm 2) adjusts memory usage with recomputation and off‑loading.

Experimental Results

End‑to‑end performance tests on Qwen 2.5 models with context lengths 32 K and 256 K show ChunkFlow achieving up to 4.53× speedup over Megatron‑LM.

Varying ChunkSize demonstrates controllable peak memory usage that depends on the preset ChunkSize rather than the longest sequence, improving robustness.

ChunkFlow now powers all Qwen models for SFT and long‑sequence CPT tasks, delivering >2× performance gains and substantial GPU cost savings.

Paper Details

Title: Efficient Long Context Fine‑tuning with Chunk Flow

Authors: Xiulong Yuan, Hongtao Xu, Wenting Shen, Ang Wang, Xiafei Qiu, Jie Zhang, Yuqiong Liu, Bowen Yu, Junyang Lin, Mingzhen Li, Weile Jia, Yong Li, Wei Lin

Link: https://arxiv.org/pdf/2503.02356