让大模型推理又快又准,自动调节长短平衡
Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
- 根据模型表现动态调整长短权衡的奖励机制
- 推理长度显著缩短,准确率基本不变
- 适合追求高效推理的部署场景
大语言模型在数学任务中展现出强大的推理能力,常通过强化学习(RL)提升。然而,训练后的模型常产生冗长的推理过程,尤其对简单问题也过度展开,导致推理成本和延迟上升。现有方法通过在奖励函数中加入固定长度惩罚来控制冗余,但此类惩罚难以调优,且无法随模型能力变化自适应,效果有限。本文提出一种自适应奖励重塑方法,使模型实现‘又快又准’的推理:当准确率高时,增加长度惩罚以加速压缩;当准确率下降时,放松惩罚以保障正确性。该机制在早期快速减少长度,后期避免过度压缩。多数据集实验表明,本方法持续且显著降低推理长度,同时基本保持准确率,为大规模语言模型的低成本自适应推理提供了新方向。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong reasoning abilities in mathematical tasks, often enhanced through reinforcement learning (RL). However, RL-trained models frequently produce unnecessarily long reasoning traces -- even for simple queries -- leading to increased inference costs and latency. While recent approaches attempt to control verbosity by adding length penalties to the reward function, these methods rely on fixed penalty terms that are hard to tune and cannot adapt as the model's reasoning capability evolves, limiting their effectiveness. In this work, we propose an adaptive reward-shaping method that enables LLMs to "think fast and right" -- producing concise outputs without sacrificing correctness. Our method dynamically adjusts the reward trade-off between accuracy and response length based on model performance: when accuracy is high, the length penalty increases to encourage faster length reduction; when accuracy drops, the penalty is relaxed to preserve correctness. This adaptive reward accelerates early-stage length reduction while avoiding over-compression in later stages. Experiments across multiple datasets show that our approach consistently and dramatically reduces reasoning length while largely maintaining accuracy, offering a new direction for cost-efficient adaptive reasoning in large-scale language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。