针对推理大模型剪枝易崩溃的问题,提出自反思剪枝框架,用模型自生成推理过程来指导剪枝。
Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models

- 用模型自身推理轨迹作为校准信号,替代人工标签
- 在20%-30%稀疏度下保持接近完整模型的准确率
- 适合需要高效部署推理大模型的资源受限场景
推理型大语言模型(如OpenAI o1、DeepSeek-R1和Qwen3)通过思维链生成实现强多步推理,但其庞大的模型规模和长推理输出导致部署成本高,不适用于资源受限环境。为降低计算与内存开销,剪枝成为可行方案,但现有方法在推理模型上表现不佳,即使20%的稀疏度也会导致性能崩溃和推理连贯性丧失。我们分析发现,现有剪枝流程失效的根本原因在于校准数据、剪枝目标与模型推理行为之间的不匹配。研究进一步表明,最可靠的校准信号来自模型自生成的推理轨迹,而非人工标注。基于此,我们提出RESP——一种自反思结构化剪枝框架,通过模型自生成校准、仅解码阶段梯度的重要性估计以及渐进式重构,使剪枝决策与推理动态对齐。在Qwen3-8B上的实验显示,RESP显著优于现有结构化剪枝方法,在GSM8K和MathQA上于20%-30%稀疏度下保持近完整模型精度,并大幅缓解高稀疏度下的性能下降。在40%稀疏度下,分别达到GSM8K 81.3%和MathQA 59.6%的准确率,超越最强基线66.87%和47%。
原文摘要 · Abstract (English)
Reasoning LLMs (RLMs) such as OpenAI o1, DeepSeek-R1, and Qwen3 deliver strong multi-step reasoning through chain-of-thought generation, but their large model sizes and lengthy decode-time outputs make them costly to deploy and unsuitable for resource-constrained settings. To reduce computing and memory cost, pruning offers a promising solution by removing unimportant parameters. However, despite their success on standard LLMs, existing pruning methods severely damage RLMs, as even moderate sparsity (e.g., 20%) can collapse accuracy and completely disrupt the model's reasoning coherence. We begin by analyzing why existing pruning pipelines fail on reasoning LLMs and find that their brittleness largely stems from a mismatch between the calibration data, the pruning objective, and the model's decode-time reasoning behavior. Our study further shows that the most reliable calibration signal comes not from human-written labels but from the model's own self-generated reasoning traces, which more accurately reflect its inference distribution. Guided by these insights, we introduce RESP, a self-reflective structured pruning framework that aligns pruning decisions with the model's reasoning dynamics through self-generated calibration, decode-only gradient-based importance estimation, and progressive regeneration that maintains calibration fidelity as sparsity increases. Experiments on Qwen3-8B demonstrate that RESP markedly outperforms existing structured pruning methods on both GSM8K and MathQA, preserving near-dense accuracy at 20-30% sparsity and substantially mitigating performance collapse at higher sparsity levels. At 40% sparsity, RESP attains 81.3% accuracy on GSM8K and 59.6% on MathQA, surpassing the strongest baselines by 66.87% and 47%, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。