用智能掩码优化加速大模型，50亿参数重训练就超密集模型。
SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask
- 通过海森重要性引导的软掩码渐进优化，直接改进稀疏结构。
- 2:4稀疏下仅用50亿重训练样本，零样本准确率达57.27%。
- 适合追求高效推理的开发者，尤其适合硬件原生支持稀疏的场景。
半结构化稀疏能利用硬件原生支持加速大语言模型，但后训练半结构化剪枝常因强结构耦合导致性能大幅下降。现有方法依赖大规模稀疏重训练恢复精度，计算成本高昂。本文提出SparseForge，一种后训练框架，通过直接优化稀疏掩码而非扩大重训练规模，提升恢复效率。该方法结合海森重要性估计与软掩码渐进退火，生成可被硬件执行的结构化稀疏模式，实现稳定高效的稀疏恢复。在LLaMA-2-7B模型上，2:4稀疏下仅使用50亿重训练样本，平均零样本准确率达57.27%，超越密集模型的56.43%，接近采用400亿样本的先进方法（57.52%）。该方法在不同模型族中均展现出一致的精度-效率优势。
原文摘要 · Abstract (English)
Semi-structured sparsity provides a practical path to accelerate large language models (LLMs) with native hardware support, but post-training semi-structured pruning often suffers from substantial quality degradation due to strong structural coupling. Existing methods rely on large-scale sparse retraining to recover accuracy, resulting in high computational cost. We propose SparseForge, a post-training framework that improves recovery efficiency by directly optimizing the sparsity mask rather than scaling up retraining tokens. SparseForge combines Hessian-aware importance estimation with progressive annealing of soft masks into hardware-executable structured sparsity, enabling stable and efficient sparse recovery. On LLaMA-2-7B under 2:4 sparsity, SparseForge achieves 57.27% average zero-shot accuracy with only $\textbf{5B}$ retraining tokens, surpassing the dense model's 56.43% accuracy and approaching the 57.52% result of a state-of-the-art method using $\textbf{40B}$ tokens. Such improvements on the accuracy-efficiency trade-off from SparseForge are shown to be consistent across model families.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。