arXiv:2605.09806cs.LGcs.AI2026-05

让大模型自动调节推理长度,又快又准

LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models

论文配图:LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models
图 1 · 摘自论文原文
  • 根据推理过程动态调整效率与正确性的平衡
  • 在5个数学题集上准确率和效率双优,输出更短
  • 适合需要高效推理的模型部署场景

大型推理模型如OpenAI o1和DeepSeek-R1随着推理能力提升,其思维链(CoT)轨迹变得越来越冗长,远超问题实际所需,造成计算、延迟和上下文预算浪费。虽然强化学习中引入长度效率奖励可缓解此问题,但现有方法面临两大挑战:正确性与效率的最优平衡在训练过程中非静态,且不同问题的内在推理预算差异巨大。依赖固定奖励权重和全局长度约束必然导致准确率下降或压缩潜力未发挥。为此,我们提出LEAD(Length-Efficient Adaptive and Dynamic reasoning),用在线自适应机制替代静态启发式策略。LEAD通过潜在缩放不稳定性,在每一步动态校准正确性与效率的权衡,将优化资源集中于最有效的学习信号。同时,它基于模型自身正确推理路径在线估计每个问题的自适应目标长度,并施加对称效率奖励,惩罚过度思考和过度压缩。在五个数学推理基准上的评估显示,LEAD在所有强化学习训练的高效推理方法中取得了最高准确率和准确率-效率得分,且输出显著短于基线模型。

原文摘要 · Abstract (English)

Large reasoning models, such as OpenAI o1 and DeepSeek-R1, tend to become increasingly verbose as their reasoning capabilities improve. These inflated Chain-of-Thought (CoT) trajectories often exceed what the underlying problems require, wasting compute, latency, and context budgets. While introducing length-based efficiency rewards during reinforcement learning offers a natural remedy, existing methods struggle with two fundamental challenges: the optimal balance between correctness and efficiency is non-stationary throughout training, and intrinsic reasoning budgets vary drastically across problems. Relying on static reward weights and global length constraints inevitably forces a compromise between degraded accuracy and unrealized compression. To overcome these limitations, we propose LEAD (Length-Efficient Adaptive and Dynamic reasoning), a method that replaces static heuristics with online, self-adaptive mechanisms. LEAD dynamically calibrates the correctness-efficiency trade-off at each step using a Potential-Scaled Instability, directing optimization capacity to the most informative learning signal. Furthermore, it estimates an adaptive per-problem target length online based on the model's own correct rollouts, applying a symmetric efficiency reward that penalizes both overthinking and over-compression. Evaluated on five mathematical reasoning benchmarks, LEAD achieves the highest accuracy and Accuracy-Efficiency Score among RL-trained efficient-reasoning methods while producing substantially shorter outputs than the base model.

推理优化强化学习效率压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。