提出连续稀疏训练方法，让2:4稀疏预训练更高效稳定。
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training
- 设计连续投影函数实现平滑稀疏化，避免传统方法的不连续问题。
- 在LLaMA-7B上达到90.1%的基线性能，接近全参模型效果。
- 适合追求高效率稀疏训练的开发者和部署场景。
深度神经网络训练成本高昂。Nvidia Ampere和Hopper GPU可通过实现2:4稀疏性使矩阵乘法速度提升至密集等价物的两倍。然而，以往基于STE的2:4稀疏预训练方法（如硬阈值STE、SR-STE）因修剪函数不连续而面临优化困难。本文系统分析了传统N:M稀疏训练的瓶颈，指出三个不连续带来的问题：下降方向错误、无法预测下降量、稀疏掩码振荡。为此，提出S-STE方法，包含两个部分：将权重连续投影至2:4稀疏，并采用每张量固定缩放因子重校准稀疏权重。同时，采用最小方差无偏估计计算激活梯度，全程使用FP8量化。实验表明，该方法超越现有2:4稀疏预训练方案，在LLaMA-7B上达到90.1%基线性能，甚至可媲美全参数模型。代码已开源：https://github.com/huyz2023/2by4-pretrain。
原文摘要 · Abstract (English)
Training deep neural networks (DNNs) is costly. Fortunately, Nvidia Ampere and Hopper GPUs can accelerate matrix multiplications twice as fast as a dense equivalent by implementing 2:4 sparsity. However, previous STE-based 2:4 pre-training methods (e.g. STE with hard-thresholding, SR-STE) suffer from optimization difficulties because of discontinuous pruning function. In this study, we comprehensively analyse the bottleneck of traditional N:M sparse training and recognize three drawbacks with discontinuity: incorrect descending direction, inability to predict the amount of descent and sparse mask oscillation. In light of this, we propose S-STE, a simple yet powerful 2:4 training method that contains two parts: to continuously project weights to be 2:4 sparse, and to rescale sparse weights with a per-tensor fixed scaling factor. Besides, we adopt minimum-variance unbiased estimation for activation gradient and FP8 quantization for whole process. Results show that our method surpasses previous 2:4 pre-training recipes and is comparable even with full parameter models. Our toolkit is available at https://github.com/huyz2023/2by4-pretrain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。