通过智能剪枝提升长思维链推理能力,让小模型更高效。
Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning
- 将长思维链转为逻辑图,按自验证约束剪除低效步骤
- 仅保留验证环节的剪枝使准确率提升且节省30%以上token
- 适合想压缩大模型推理能力到小模型的研究者
长思维链(Long-CoT)可提升大模型的推理准确率,但其冗长自省风格不利于压缩至小语言模型(SLMs)。本文从能力对齐视角重新审视长思维链压缩,提出结构感知的Prune-on-Logic框架:将长思维链转化为逻辑图,并在自验证约束下选择性剪除低效推理步骤。系统分析三种剪枝策略——整链、核心推理与验证剪枝后发现,仅验证剪枝能持续提升准确率并减少token使用,而剪除推理步骤或盲目剪枝会降低性能。研究揭示,有效剪枝应使监督信号与模型容量对齐,而非简单缩短输入。该效果在多种任务、模型规模和思维链能力下均成立,大模型因推理更冗余而受益更多。实证表明,剪枝是实现思维链与小模型能力匹配的结构性优化策略。
原文摘要 · Abstract (English)
Long chain-of-thought (Long-CoT) reasoning improves accuracy in LLMs, yet its verbose, self-reflective style often hinders effective distillation into small language models (SLMs). We revisit Long-CoT compression through the lens of capability alignment and ask: Can pruning improve reasoning? We propose Prune-on-Logic, a structure-aware framework that transforms Long-CoT into logic graphs and selectively prunes low-utility reasoning steps under self-verification constraints. Through systematic analysis across three pruning strategies targeting entire chains, core reasoning, and verification, we find that verification pruning consistently improves accuracy while reducing token usage, whereas pruning reasoning steps or indiscriminate pruning degrades performance. Our study reveals that effective pruning aligns supervision with model capacity rather than merely shortening inputs. Gains hold across tasks, model scales, and CoT capability, with larger models benefiting more from pruning due to richer but more redundant reasoning. Our empirical findings highlight pruning as a structural optimization strategy for aligning CoT reasoning with SLM capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。