通过诱导稀疏性，实现大模型高效剪枝而不损失性能
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
- 先通过数学变换提升权重分布稀疏性，再用谱范数损失增强特征稀疏性
- 在多个模型和任务上实现更高稀疏率下的性能保持，优于现有方法
- 适合需要压缩大模型但不想牺牲精度的研究者与工程师
大型语言模型在文本生成方面表现出色，但参数规模增大带来了计算和内存效率的挑战。后训练稀疏化（PTS）通过移除密集网络中的权重来降低模型开销，是一种有效方法。然而，原始密集矩阵本身稀疏度低，直接移除权重会破坏模型状态，即使经过后调优也难以恢复性能。本文提出「稀疏性诱导」（Sparsity Induction），在分布和特征两个层面提前促进模型向更高稀疏性演进，以突破PTS的极限。在分布层面，采用数学等价的缩放变换提升分布稀疏性，完全可吸收，不增加参数或推理开销；在特征层面，引入谱范数损失，从低秩视角推动特征稀疏性。在多种模型架构和任务上的实验表明，该方法显著增强了稀疏友好性，在剪枝性能上优于现有方法。
原文摘要 · Abstract (English)
Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our method further enhances sparsity-friendliness, achieving superior pruning performance over existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。