提出新方法提升2:4稀疏模型精度，适配大模型推理
A Proximal Operator for Inducing 2:4-Sparsity
- 基于局部特征相关性设计正则化项，优化稀疏掩码
- 在130亿参数以下模型上超越现有最优算法
- 适用于700亿参数大模型，保持与顶尖方法相当性能
AI加速器和GPU的最新进展使得高效计算稀疏矩阵乘法成为可能，尤其是当每4个连续权重中有2个为零时（即2:4稀疏）。然而这种稀疏化通常导致模型精度下降。本文推导了一种正则化项，利用特征的局部相关性，在训练好的模型中寻找更优的稀疏掩码。通过联合最小化该正则化项与局部平方损失，我们推导出对应的近端算子，并证明其在2:4稀疏情况下存在高效解。优化掩码后，采用掩码梯度更新进一步最小化局部平方损失。我们在小规模问题上验证方法有效性，并将其应用于700亿参数的大语言模型剪枝。在130亿参数以下模型上优于现有最先进算法，而在700亿参数模型上达到同等水平。
原文摘要 · Abstract (English)
Recent hardware advancements in AI Accelerators and GPUs allow to efficiently compute sparse matrix multiplications, especially when 2 out of 4 consecutive weights are set to zero. However, this so-called 2:4 sparsity usually comes at a decreased accuracy of the model. We derive a regularizer that exploits the local correlation of features to find better sparsity masks in trained models. We minimize the regularizer jointly with a local squared loss by deriving the proximal operator for which we show that it has an efficient solution in the 2:4-sparse case. After optimizing the mask, we use maskedgradient updates to further minimize the local squared loss. We illustrate our method on toy problems and apply it to pruning entire large language models up to 70B parameters. On models up to 13B we improve over previous state of the art algorithms, whilst on 70B models we match their performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。