BigMac打破多模态大模型训练的算力与内存困局，实现两者同时优化。
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
- 将编码器与生成器嵌入原有大模型训练流程，形成安全依赖的嵌套结构。
- 激活内存复杂度降至O(1)，训练速度提升1.08至1.9倍，批量增大时内存稳定。
- 适合大规模多模态模型训练，尤其对资源受限场景有显著优势。
多模态大语言模型（MLLM）训练面临模型与数据异构性的双重挑战。现有系统虽重构训练流程以应对，但仍受限于算力与内存效率之间的帕累托前沿——优化一方必牺牲另一方。本文提出BigMac，一种新型多模态大模型训练流水线。其核心思想是将编码器与生成器计算优雅地嵌入原始大模型训练流程，构建依赖安全的嵌套结构。该设计使编码器与生成器的激活内存复杂度降至O(1)，同时保持大模型自身激活内存复杂度不变。在计算效率上，达到理想无限内存设定下的表现。结果表明，BigMac突破了算力与内存间的帕累托前沿，实现二者同时优化。我们在多个MLLM及训练负载上评估，实验显示相比基线系统，其训练速度提升1.08×–1.9×，且在批量增大的情况下内存使用保持稳定。
原文摘要 · Abstract (English)
Training multimodal large language models (MLLMs) is challenged by both model and data heterogeneity. Existing systems redesign the training pipeline to address these challenges, but remain bound by a Pareto frontier between compute and memory efficiency, improving one only at the expense of the other. We present BigMac, a new training pipeline for multimodal LLMs. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure. With this design, BigMac reduces the activation memory complexity of the encoder and generator to O(1) while keeping the activation memory complexity of the LLM unchanged. At the same time, it achieves the same computational efficiency as the idealized setting with unlimited memory. As a result, BigMac breaks the Pareto frontier between computational efficiency and memory usage, enabling simultaneous optimization of both computation and memory in MLLM training. We evaluate BigMac on multiple MLLMs and training workloads. Experimental results show that BigMac achieves a 1.08$\times$-1.9$\times$ training speedup over baseline systems while maintaining stable memory usage as batch size increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。