针对低秩瓶颈模型训练慢问题，提出高效并行框架BOOST
BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
- 设计感知瓶颈的张量并行机制，优化通信与计算
- 在多种架构上实现1.46至2.27倍加速，提升显卡利用率
- 适合大规模低秩大模型训练，显著降低通信开销
Transformer模型预训练规模受限于不断增长的计算和通信成本。低秩瓶颈架构能在最小影响精度的前提下显著减少训练时间和内存占用。然而，现有瓶颈架构在标准张量并行下扩展性差，直接套用为全秩设计的3D并行会导致过度通信和显卡利用率低下。为此，我们提出BOOST——一个专为大规模低秩瓶颈架构设计的高效训练框架。BOOST引入新型瓶颈感知张量并行，并结合在线RMSNorm、线性层分组和低秩激活检查点等优化，实现端到端训练加速。在不同低秩瓶颈架构上的评估表明，相比全秩基线模型，BOOST实现1.46-1.91×加速；相比未经优化的低秩3D并行模型，加速比达1.87-2.27×，同时提升显卡利用率并降低通信开销。
原文摘要 · Abstract (English)
The scale of transformer model pre-training is constrained by the increasing computation and communication cost. Low-rank bottleneck architectures offer a promising solution to significantly reduce the training time and memory footprint with minimum impact on accuracy. Despite algorithmic efficiency, bottleneck architectures scale poorly under standard tensor parallelism. Simply applying 3D parallelism designed for full-rank methods leads to excessive communication and poor GPU utilization. To address this limitation, we propose BOOST, an efficient training framework tailored for large-scale low-rank bottleneck architectures. BOOST introduces a novel Bottleneck-aware Tensor Parallelism, and combines optimizations such as online-RMSNorm, linear layer grouping, and low-rank activation checkpointing to achieve end-to-end training speedup. Evaluations on different low-rank bottleneck architectures demonstrate that BOOST achieves 1.46-1.91$\times$ speedup over full-rank model baselines and 1.87-2.27$\times$ speedup over low-rank model with naively integrated 3D parallelism, with improved GPU utilization and reduced communication overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。