How BigMac Breaks the Memory‑Throughput Trade‑off in Multimodal LLM Training

Problem

Multimodal large language model (MLLM) training must balance throughput and GPU memory. A slow modality micro‑batch can stall downstream LLM pipeline stages, while keeping modality activations until gradients are computed inflates memory usage. This creates a Pareto frontier where only one of throughput or memory can be optimized.

Conventional Designs

Compute‑efficient pipeline : Encoder and generator are decoupled from the LLM pipeline (e.g., LongCat‑Flash‑Omni modality‑decoupled parallelism). LLM stages follow a 1F1B or interleaved 1F1B schedule, preserving throughput, but activations from the encoder (and possibly generator) must be retained for the entire LLM forward‑backward pass, leading to high memory consumption.

Memory‑efficient pipeline : Encoder and generator are merged into the same pipeline as the LLM (e.g., Megatron‑Core multimodal example). Activation lifetimes are shortened, reducing memory, but any slow modality micro‑batch forces the entire LLM pipeline to wait, creating bubbles and reducing compute efficiency.

BigMac Design

BigMac introduces a dependency‑safe nested pipeline that keeps the mature LLM pipeline schedule as the backbone and inserts encoder and generator computations at points where inputs are ready and the LLM execution order is not disturbed. This preserves LLM throughput while bounding modality activation memory to O(1).

Scheduler and Executor

The runtime consists of a Scheduler that generates a global operator table covering all ranks, micro‑batches, and module types (LLM forward/backward, encoder forward/backward, generator forward/backward, and required communication). An Executor interprets the per‑rank operator sequence and dispatches each operator to the appropriate backend: LLM operators to Megatron Core, modality operators to their own runtimes, and data movement to the network layer. This makes the schedule visible and checkable while remaining backend‑friendly.

Pipeline‑parallelism‑transparent Interface

Algorithm engineers declare only module inputs and outputs. BigMac automatically places modules in the global schedule, handles activation handoff, gradient handoff, and cross‑device communication, allowing a model verified in single‑card or data‑parallel mode to scale to pipeline‑parallel training without manual stage‑division code.

Schedule‑aware Toolchain

BigMac provides:

A profiler that records per‑operator execution times and exports per‑rank traces, enabling inspection of encoder/LLM/generator interactions and identification of bubbles caused by compute imbalance, communication waits, or sub‑optimal micro‑batch grouping.

A simulator that evaluates alternative pipeline‑parallel (PP) / virtual pipeline parallel (VPP) configurations, micro‑batch counts, and module placements before launching expensive full‑scale runs.

A visualizer that renders the global schedule and operator‑level timelines for debugging.

Experimental Evaluation

Two representative workloads were used:

MLLM‑Understanding : Qwen3‑30B‑A3B backbone + 1.3 B ViT encoder. Compared to the compute‑efficient baseline Optimus, BigMac achieved 1.08×–1.10× faster training. Compared to the memory‑efficient baseline Megatron‑DistTrain, it achieved 1.6×–1.9× speedup. Peak GPU memory remained flat as global batch size increased, whereas Optimus’ memory grew rapidly and eventually OOM.

MLLM‑Generation : Same backbone and encoder plus a 20 B parameter MMDiT generator. Optimus OOM on all tested batch sizes. BigMac avoided OOM by running generator backward as soon as gradients were ready, releasing generator activations early. Compared to Megatron‑DistTrain, BigMac was 1.5×–1.9× faster while keeping memory stable.

Conclusion

BigMac breaks the compute‑memory Pareto frontier by nesting modality computation within a stable LLM pipeline, exposing a global schedule that is visible, verifiable, and executable, and providing a PP‑transparent interface together with schedule‑aware diagnostics. This enables scalable multimodal LLM training without sacrificing throughput or exceeding GPU memory limits.

Paper : https://arxiv.org/pdf/2605.25451 Code :

https://github.com/Dots-Infra/BigMac/