用少量数据让大模型推理更短更高效，不丢准确率。
Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
- 通过难度评估筛选推理路径，减少冗余思考。
- 在多个基准上使输出长度减少超50%，性能不变。
- 适合追求高效推理的部署场景，尤其资源受限时。
大型推理模型（LRM）通过长链式思维（CoT）在复杂任务中表现优异，但过长的输出增加了计算成本，可能引发过度思考。现有方法常需大量资源或牺牲推理质量。本文研究如何在有限调优条件下缩短LRM生成长度。分析生成路径分布并基于难度估计过滤轨迹；进一步在统一的Bradley-Terry损失框架下研究不同偏好优化目标的收敛特性。据此提出长度可控偏好优化（LCPO），直接平衡与NLL损失相关的隐式奖励。实验表明，该方法在多个基准上平均输出长度减少超过50%，同时保持推理性能。工作凸显了轻量级高效引导策略在提升推理效率方面的潜力。
原文摘要 · Abstract (English)
Recent advances in Large Reasoning Models (LRMs) have demonstrated strong performance on complex tasks through long Chain-of-Thought (CoT) reasoning. However, their lengthy outputs increase computational costs and may lead to overthinking, raising challenges in balancing reasoning effectiveness and efficiency. Current solutions often compromise reasoning quality or require extensive resources. In this paper, we investigate how to reduce the generation length of LRMs with limited tuning. We analyze generation path distributions and filter generated trajectories through difficulty estimation. Subsequently, we analyze the convergence characteristics of various preference optimization objectives under a unified Bradley-Terry loss based framework. Based on the analysis, we propose Length Controlled Preference Optimization (LCPO) that directly balances the implicit reward related to NLL loss. LCPO can effectively learn length preference with limited data and training. Extensive experiments demonstrate that our method significantly reduces the average output length of LRMs by over 50\% across multiple benchmarks while maintaining the reasoning performance. Our work highlights the potential for computationally efficient approaches in guiding LRMs toward efficient reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。