让推理模型剪枝更准：用思维链重建提升压缩效果
Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction
- 剪枝时同时重建输入和思维链激活，提升推理性能
- 传统剪枝使推理变慢且效果差，新方法显著改善
- 可直接接入现有剪枝流程，适合部署推理模型的团队
推理类语言模型（如 DeepSeek-R1）在推理时会产生长思维链，导致大规模部署成本高。我们发现，使用神经网络剪枝等压缩技术时，性能损失比常规语言建模任务更大，甚至可能使模型变慢，因为剪枝后模型生成更多思考令牌但质量下降。这主要是因为标准剪枝方法侧重输入重建，而推理是解码主导的任务。为此，我们提出一种简单有效的改进：在剪枝过程中，联合重建输入和模型的在线思维链激活。该方法称为「推理感知压缩」（RAC），可无缝集成至 SparseGPT 等现有剪枝流程，并显著提升性能。代码已公开于 https://github.com/RyanLucas3/RAC。
原文摘要 · Abstract (English)
Reasoning language models such as DeepSeek-R1 produce long chain-of-thought traces during inference time which make them costly to deploy at scale. We show that using compression techniques such as neural network pruning produces greater performance loss than in typical language modeling tasks, and in some cases can make the model slower since they cause the model to produce more thinking tokens but with worse performance. We show that this is partly due to the fact that standard LLM pruning methods often focus on input reconstruction, whereas reasoning is a decode-dominated task. We introduce a simple, drop-in fix: during pruning we jointly reconstruct activations from the input and the model's on-policy chain-of-thought traces. This "Reasoning-Aware Compression" (RAC) integrates seamlessly into existing pruning workflows such as SparseGPT, and boosts their performance significantly. Code reproducing the results in the paper can be found at: https://github.com/RyanLucas3/RAC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。