Xiaohongshu Unveils BigMac: Breaking the Memory-Speed Trade-off in Multi-modal Training

Multi-modal large language models are emerging as the cornerstone of next-generation AI, yet their training infrastructure has long been hindered by a persistent trade-off: systems optimized for speed often struggle with memory constraints, while those prioritizing memory efficiency tend to be slow. The Dots Infra team at Xiaohongshu identifies this bottleneck as the Pareto frontier of multi-modal pipeline training and has now broken through it. On July 22, the team open-sourced BigMac, a novel pipeline parallel training paradigm designed specifically for native multi-modal scenarios. The project is now available on GitHub under the Dots-Infra repository.

Unlike standard transformers, multi-modal models are inherently complex. A typical Multi-modal Large Language Model (MLLM) comprises three distinct components: modal encoders that convert images and audio into embeddings, an LLM backbone for inference, and a generator that maps LLM outputs back to target modalities such as images or speech. The significant structural differences between these components create substantial challenges when integrating them into a single training pipeline.

Current industry solutions generally fall into two categories. The first approach prioritizes computational efficiency by decoupling encoders and generators from the LLM pipeline, running them separately. This prevents fluctuations in modal module duration from creating bubbles in the LLM pipeline, but it causes activation memory to scale with the number of microbatches, leading to high costs at scale. The second approach focuses on memory efficiency by keeping all modules within the same pipeline, which shortens activation lifecycles and reduces memory usage. However, if any encoder or generator is slow, the entire LLM pipeline stalls, resulting in tail bubbles. As model scale increases, both designs reveal critical bottlenecks.

BigMac addresses this challenge with a straightforward yet pivotal principle: the core LLM pipeline remains the primary focus. Large-scale LLM training already relies on mature scheduling strategies like 1F1B or interleaved 1F1B, which are deeply integrated into production-grade training stacks. Rather than replacing these established methods, BigMac uses the LLM schedule as its underlying timeline, strategically inserting encoder and generator computations when inputs are ready and without disrupting the LLM execution order. The team refers to this architecture as a "quasi-dependency-safe nested pipeline."

This design delivers two key advantages. First, fluctuations in encoder and generator duration no longer propagate through the LLM pipeline, allowing the LLM to maintain its optimal pace. Second, memory requirements for modal activations are reduced to O(1) at the algorithmic level. Encoders no longer need to retain activations for all microbatches until the pipeline concludes, and generators avoid extending long activation tails. Essentially, BigMac does not trade memory for performance bubbles; instead, it restructures the scheduling logic to make these two objectives compatible rather than mutually exclusive.

Bridging the gap between schedule design and actual execution involves significant engineering hurdles, including system integration, interface definition, and performance debugging. BigMac is engineered to address these specific challenges by providing three core capabilities. First, it makes the global schedule transparent: the scheduler generates a comprehensive operator table covering all pipeline ranks, microbatches, and module types at runtime. The executor then distributes these into local sequences for each rank, integrating seamlessly with LLM backends like Megatron Core, modal runtimes, and communication layers. This visibility allows for easier checking and backend optimization.

Second, BigMac offers a pipeline parallel interface that remains invisible to algorithm engineers. Users only need to define what each module produces and consumes; the system handles stage division, activation and gradient handoff, and cross-device communication internally. This allows multi-modal experiments verified on a single GPU to scale naturally to pipeline parallelism. Third, it provides a suite of profiling, simulation, and visualization tools. These tools break down training iterations to the operator level, revealing exactly what each rank is doing at every time step, identifying idle periods, and highlighting dependency bottlenecks. The simulator also allows teams to test various PP configurations and microbatch combinations before committing to full-scale training, estimating their impact on bubbles and throughput.

The effectiveness of BigMac has been validated across two representative workloads. In MLLM-Understanding tasks, using Qwen3-30B-A3B as the backbone and a 1.3B ViT encoder, BigMac achieves a 1.08x to 1.1x speedup compared to the computationally efficient baseline Optimus, and a 1.6x to 1.9x speedup over the memory-efficient baseline Megatron-DistTrain. Crucially, memory usage remains stable as the per-GPU batch size increases, whereas Optimus experiences peak memory surges that eventually lead to Out-of-Memory (OOM) errors at larger batches. MLLM-Generation tasks are more complex, involving a 20B MMDiT generator, where the performance gap widens further: Optimus fails across all tested batch sizes due to OOM, while BigMac avoids this by efficiently executing the generator backward and rapidly releasing activations. Compared to Megatron-DistTrain, BigMac still delivers a 1.5x to 1.9x speedup with stable memory usage. This highlights the primary value of nested pipelines: serving both encoder and generator dependencies without excessive activation retention or significant idle time.

BigMac is now a core component of Dots’ multi-modal model training infrastructure and operates within Xiaohongshu’s production environment. The accompanying paper, "BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training," is available on arXiv, along with interactive PP Profiler trace examples. For researchers and engineers struggling with the memory-speed trade-off in multi-modal training, this production-validated open-source project offers a practical solution that saves significant time and effort.