通过技能感知分解与蒸馏剪枝，让大模型推理更高效且不丢准确率。
DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models
- 用教师模型做技能感知的步骤拆解与内容剪枝，再蒸馏给学生模型。
- 在GSM8K上推理token减少64%，准确率反而提升至94.1%。
- 适合追求高效推理的数学题求解场景，尤其关注推理压缩的开发者。
大型推理模型（LRMs）虽在复杂推理任务中表现优异，但其推理过程常伴随冗长的思维链（CoT），导致效率低下。为此，我们提出一种融合推理时剪枝与基于训练的蒸馏的混合框架——蒸馏推理剪枝（DRP）。DRP利用教师模型进行技能感知的步骤分解与内容剪枝，并将剪裁后的推理路径蒸馏至学生模型，使其既能高效又能准确推理。在多个挑战性数学推理数据集上，使用DRP训练的模型显著提升了词元效率，未牺牲准确性：在GSM8K上，平均词元使用量从917降至328，准确率由91.7%提升至94.1%；在AIME上实现43%的词元减少，性能无下降。进一步分析表明，训练思维链的结构与学生模型推理能力对齐，是有效知识迁移和性能提升的关键。
原文摘要 · Abstract (English)
While Large Reasoning Models (LRMs) have demonstrated success in complex reasoning tasks through long chain-of-thought (CoT) reasoning, their inference often involves excessively verbose reasoning traces, resulting in substantial inefficiency. To address this, we propose Distilled Reasoning Pruning (DRP), a hybrid framework that combines inference-time pruning with tuning-based distillation, two widely used strategies for efficient reasoning. DRP uses a teacher model to perform skill-aware step decomposition and content pruning, and then distills the pruned reasoning paths into a student model, enabling it to reason both efficiently and accurately. Across several challenging mathematical reasoning datasets, we find that models trained with DRP achieve substantial improvements in token efficiency without sacrificing accuracy. Specifically, DRP reduces average token usage on GSM8K from 917 to 328 while improving accuracy from 91.7% to 94.1%, and achieves a 43% token reduction on AIME with no performance drop. Further analysis shows that aligning the reasoning structure of training CoTs with the student's reasoning capacity is critical for effective knowledge transfer and performance gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。