BigMac: Breaking the Pareto Frontier of Compute‑Memory Trade‑offs in Multimodal LLM Training

Training multimodal large language models (MLLMs) requires coordinating heterogeneous modules—modal encoders, an LLM backbone, and modal generators—leading to a classic trade‑off between compute efficiency (fast pipelines but high activation memory) and memory efficiency (low memory but frequent pipeline bubbles).

BigMac resolves this Pareto dilemma by keeping the LLM pipeline schedule as the immutable backbone and inserting encoder and generator computations at dependency‑safe points that do not disrupt the LLM order. This nested pipeline reduces encoder and generator activation memory to O(1) while preserving the high‑throughput schedule of the LLM.

The system consists of a Scheduler that emits a global operator table covering every rank, micro‑batch, and module type, and an Executor that dispatches each operator to the appropriate backend (Megatron Core for LLM, dedicated runtimes for encoder/generator, and a communication layer). Making the schedule explicit enables developers to inspect, verify, and modify the global plan without rewriting runtime code.

BigMac provides a pipeline‑parallelism‑transparent interface: algorithm engineers declare only the inputs and outputs of each module, and the framework automatically handles stage placement, activation/gradient handoff, and cross‑device communication, allowing single‑card or data‑parallel experiments to scale to full pipeline parallelism.

A schedule‑aware toolchain complements the design. The profiler records per‑operator execution times and generates per‑rank timelines, exposing where bubbles arise (compute imbalance, communication stalls, or handoff delays). A simulator lets engineers explore alternative PP/VPP configurations, micro‑batch counts, and module placements quickly, guiding performance tuning before launching costly training runs.

Evaluation on two representative workloads demonstrates the benefits. On the MLLM‑Understanding task (Qwen3‑30B‑A3B backbone with a 1.3 B ViT encoder), BigMac achieves 1.08‑1.1× speedup over the compute‑efficient Optimus baseline and 1.6‑1.9× over the memory‑efficient Megatron‑DistTrain baseline, while keeping peak memory stable as batch size grows. On the MLLM‑Generation task (adding a 20 B MMDiT generator), BigMac delivers 1.5‑1.9× speedup and avoids the out‑of‑memory failures that affect both baselines.

By breaking the compute‑memory Pareto frontier, exposing a visible schedule, and offering developer‑friendly interfaces and diagnostics, BigMac enables scalable, efficient multimodal LLM training and is available as an open‑source project at https://github.com/Dots-Infra/BigMac/.