提出可学习的稀疏掩码方法，让大模型更高效且不损失性能。
ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
- 用正则化优化替代传统启发式规则，实现全局优化掩码选择。
- 在7个主流模型上均显著优于已有方法，精度损失小、加速效果好。
- 适合追求模型轻量化部署的研究者与工程师使用。
大规模语言模型在自然语言处理任务中表现优异，但其巨大规模导致推理效率低、成本高。半结构化剪枝成为提升模型效率的有效手段，但现有方法多依赖局部、逐层的启发式规则，未能利用全局反馈信息。本文提出 ProxSparse，一种基于学习的掩码选择框架，通过正则化优化将原本刚性且不可导的掩码选择过程转化为平滑优化过程，支持渐进式掩码探索。一旦掩码确定，无需额外权重更新。在7个广泛使用的模型上进行的大量实验表明，ProxSparse持续优于以往提出的半结构化掩码选择方法，显著提升了剪枝效果，验证了所提学习方法在半结构化剪枝中的有效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。