提出衡量推理效率的新标准,让大模型在保持准确率的同时大幅缩短推理过程。
How Far Are We from Optimal Reasoning Efficiency?
- 构建推理效率前沿,量化最优推理表现的理论上限。
- 发现现有方法在短文本与高精度间难以兼顾,存在显著效率差距。
- 设计强化学习算法REO-RL,实现50%以上效率提升,适合部署优化场景。
大型推理模型(LRMs)通过扩展的思维链(CoT)展现出强大解题能力,但常产生冗长重复的推理过程,导致推理成本高昂,限制实际应用。现有微调方法虽旨在提升效率,却因评估不一致而难以衡量成效。本文提出推理效率前沿,即通过不同微调策略和训练配置对基础模型进行微调所得到的经验上界。基于此,我们引入推理效率差距(REG),统一量化任意微调模型与该前沿的偏离程度。在多个挑战性数学基准上的系统评估显示,当前方法或以牺牲准确性换取简短输出,或在严格令牌预算下仍显低效。为缩小效率差距,我们提出REO-RL,一类通过聚焦稀疏令牌预算集进行强化学习的算法。利用在精心选择的预算点上的数值积分,REO-RL以极小计算量近似完整效率目标。实验表明,所提指标REG能有效捕捉准确率与长度之间的权衡;低REG方法可在保持准确率前提下显著压缩输出长度。我们的方法在所有测试模型上效率差距减少≥50%,且在16K令牌预算下达到Qwen3-4B/8B的效率前沿,仅伴随轻微准确率损失。消融实验验证了指数型预算策略的有效性。最终发现,使模型完美对齐效率前沿仍是未解挑战。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) demonstrate remarkable problem-solving capabilities through extended Chain-of-Thought (CoT) reasoning but often produce excessively verbose and redundant reasoning traces. This inefficiency incurs high inference costs and limits practical deployment. While existing fine-tuning methods aim to improve reasoning efficiency, assessing their efficiency gains remains challenging due to inconsistent evaluations. In this work, we introduce the reasoning efficiency frontiers, empirical upper bounds derived from fine-tuning base LRMs across diverse approaches and training configurations. Based on these frontiers, we propose the Reasoning Efficiency Gap (REG), a unified metric quantifying deviations of any fine-tuned LRMs from these frontiers. Systematic evaluation on challenging mathematical benchmarks reveals significant gaps in current methods: they either sacrifice accuracy for short length or still remain inefficient under tight token budgets. To reduce the efficiency gap, we propose REO-RL, a class of Reinforcement Learning algorithms that minimizes REG by targeting a sparse set of token budgets. Leveraging numerical integration over strategically selected budgets, REO-RL approximates the full efficiency objective with low error using a small set of token budgets. Through systematic benchmarking, we demonstrate that our efficiency metric, REG, effectively captures the accuracy-length trade-off, with low-REG methods reducing length while maintaining accuracy. Our approach, REO-RL, consistently reduces REG by >=50 across all evaluated LRMs and matching Qwen3-4B/8B efficiency frontiers under a 16K token budget with minimal accuracy loss. Ablation studies confirm the effectiveness of our exponential token budget strategy. Finally, our findings highlight that fine-tuning LRMs to perfectly align with the efficiency frontiers remains an open challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。