通过神经元级稀疏激活加速大模型预训练，提升1.4至1.7倍速度。
To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
- 采用2:4权值稀疏与v:n:m激活稀疏，加速FFN中全部矩阵乘法。
- 稀疏训练结合密集微调，实现性能无损且整体训练提速1.4~1.7倍。
- 兼容A100及以上显卡，适配量化与专家混合模型架构。
大语言模型的训练通常受制于矩阵乘法的计算瓶颈。在Transformer结构中，前馈网络（FFN）占用了大量矩阵运算，对更大模型而言，该部分可高达总预训练浮点操作数的50%。本文提出利用硬件支持的稀疏性，对FFN中的所有矩阵乘法进行加速，采用2:4权值稀疏和v:n:m（Venom）激活稀疏策略。我们的方法包含稀疏训练阶段以加速大部分预训练过程，并在后期辅以常规密集训练步骤。实验表明，使用该方法训练的模型在质量基准测试中表现相当，整体训练速度提升1.4至1.7倍。该方案适用于所有从A100开始的NVIDIA GPU，且与量化等常见优化技术正交，也可应用于混合专家模型架构。
原文摘要 · Abstract (English)
Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Forward Network (FFN), and this portion increases for larger models, up to 50% of the total pretraining floating point operations. We show that we can leverage hardware-accelerated sparsity to accelerate all matrix multiplications in the FFN, with 2:4 sparsity for weights and v:n:m (Venom) sparsity for activations. Our recipe relies on sparse training steps to accelerate a large part of the pretraining, associated with regular dense training steps towards the end. Overall, models trained with this approach exhibit the same performance on our quality benchmarks, and can speed up training end-to-end by 1.4 to 1.7x. This approach is applicable to all NVIDIA GPUs starting with the A100 generation, and is orthogonal to common optimization techniques, such as, quantization, and can also be applied to mixture-of-experts model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。