研究大模型压缩对推理能力的影响,发现关键权重保护可显著提升性能。
When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models
- 通过激活分析定位压缩对推理的关键影响权重。
- 仅保护2%过度压缩权重,准确率提升6.57%。
- 揭示模型压缩中知识记忆与推理的差异性影响,适合模型优化者参考。
压缩技术(量化、蒸馏、剪枝)能提升大推理模型(LRMs)的计算效率,但现有研究或未能全面比较三类方法,或缺乏深层机制分析。本文通过性能基准测试与机理解释,评估量化、蒸馏、剪枝的DeepSeek-R1模型在四个推理数据集(AIME 2024、FOLIO、Temporal Sequences、MuSiQue)上的表现。采用均值差与归因修补技术,聚焦压缩后线性组件的激活,解析权重与各类推理能力之间的细粒度因果关系。结果表明:动态量化至2.51比特的R1模型接近原版性能;核心发现包括:(1)权重数量对知识记忆的影响大于推理,剪枝与蒸馏风险更高;(2)蒸馏模型最后一层MLP上投影是关键组件;(3)当前量化过度压缩末层模块与MLP门控投影,仅保护2%被过度压缩权重,平均准确率提升6.57%,远超现有最优方法。
原文摘要 · Abstract (English)
Compression methods, including quantization, distillation, and pruning, improve the computational efficiency of large reasoning models (LRMs). However, existing studies either fail to sufficiently compare all three compression methods on LRMs or lack in-depth interpretation analysis. In this paper, we investigate how the reasoning capabilities of LRMs are compromised during compression, through performance benchmarking and mechanistic interpretation. To uncover the effects of compression on reasoning performance, we benchmark quantized, distilled, and pruned DeepSeek-R1 models on four reasoning datasets (AIME 2024, FOLIO, Temporal Sequences, and MuSiQue). To precisely locate compression effects on model weights, we adapt difference of means and attribution patching techniques, focusing on the activation of every linear component in compressed LRMs, to interpret fine-grained causal relationships between weights and various reasoning capabilities. This fine-grained interpretation addresses a fundamental question of compression: which weights are the most important for reasoning? Overall, we find dynamically quantized 2.51-bit R1 reaches close-to-R1 performance. With empirical verification, we present three main findings that generalize across both R1 and non-R1 LRMs: (1) Weight count has a greater impact on LRMs' knowledge memorization than reasoning, highlighting the risks of pruning and distillation; (2) The MLP up projection in the final layer of distilled LRMs is one of the most important components, offering a new perspective on locating critical weights - a fundamental problem in model compression; and (3) Current quantization methods overly compress the final-layer modules and MLP gate projections, so protecting just 2% of all weights that are excessively compressed can raise average accuracy by 6.57%, greatly surpassing the state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。