精心剪枝反而能提升大模型推理性能,突破传统认知。
Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling

- 采用细粒度权重剪枝,而非整块层删除
- 在4个推理基准上优于结构化剪枝,部分超越原始模型
- 适合追求高效推理的开发者与部署研究者
大语言模型通过测试时计算扩展(TTS)展现出强大的推理能力,在数学与编程基准上表现优异。与此同时,模型压缩领域的剪枝方法致力于移除冗余或有害参数而不损失任务性能。本文聚焦于推理类LLM,重新审视结构化剪枝(移除整个层块)会显著损害TTS性能这一既有结论。我们发现,采用非结构化剪枝(仅精细删除特定冗余权重)在s1.1-7B和Qwen3-8B两个推理模型上,于四个推理基准上均表现出对结构化剪枝的优越性,甚至在某些情况下超越未剪枝的完整模型。此外,我们还实证研究了不同层间稀疏度分配策略的影响,这些策略是实现非结构化剪枝的关键参数选择。结果挑战了‘剪枝必然降低TTS性能’的传统观点,表明经过精心设计的剪枝可有效保持甚至增强模型的测试时推理能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) now exhibit remarkable reasoning capabilities through test-time compute scaling (TTS), with impressive performance across math and coding benchmarks. In parallel, research in model compression has developed pruning methods that seek to remove redundant/detrimental parameters without sacrificing task performance. The intersection of these two research advancements lays the foundation for our work. Specific to reasoning LLMs, prior work has shown that structured pruning (methods which remove entire set of layer blocks), significantly degrades TTS reasoning performance. However, in this work, we revisit this assumption and investigate whether unstructured pruning (methods that carefully remove only certain redundant/detrimental weights) exhibits similar limitations. Surprisingly, our extensive experiments across four reasoning benchmarks on two reasoning LLMs: s1.1-7B and Qwen3-8B, consistently show that unstructured pruning augments TTS performance compared to structured pruning, and at times can even outperform the unpruned full-weight LLMs. Furthermore, we also empirically study the impact of different layer-wise sparsity allocation strategies, which are an important parametric choice for instantiating these unstructured methods. These findings challenge the conventional notion that pruning always reduces TTS performance and in fact, suggest that carefully undertaken pruning can retain TTS effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。