用强化学习让大模型按任务难易动态分配计算资源，提升推理效率。
Training Language Models to Reason Efficiently
- 通过强化学习训练模型根据任务复杂度动态调整推理计算量。
- 在保持高准确率的前提下，推理成本显著降低。
- 仅需调节一个超参数即可控制模型的效率与精度平衡，适合部署优化。
扩大模型规模和训练数据虽推动了大语言模型（LLMs）性能提升，但边际收益递减，亟需新方法增强模型能力，尤其在需要高级推理的任务上。现有大型推理模型依赖长链式思维（chain-of-thoughts），虽显著提升解题能力，但带来高昂的部署成本，源于更长的生成耗时。降低推理成本对模型的经济可行性、用户体验及环境可持续性至关重要。本文提出训练大推理模型以高效推理为目标。具体而言，我们采用强化学习（RL）训练模型在推理时动态分配计算资源，依据任务复杂度进行智能调度。该方法激励模型最小化不必要的计算开销，同时保持高准确性，实现显著的效率提升。由此可衍生出一组具备不同效率层级的推理模型，其性能由单一超参数调控。在两个开源大推理模型上的实验表明，该方法在几乎不损失准确率的情况下，大幅降低推理成本。
原文摘要 · Abstract (English)
Scaling model size and training data has led to great advances in the performance of Large Language Models (LLMs). However, the diminishing returns of this approach necessitate alternative methods to improve model capabilities, particularly in tasks requiring advanced reasoning. Large reasoning models, which leverage long chain-of-thoughts, bring unprecedented breakthroughs in problem-solving capabilities but at a substantial deployment cost associated to longer generations. Reducing inference costs is crucial for the economic feasibility, user experience, and environmental sustainability of these models. In this work, we propose to train large reasoning models to reason efficiently. More precisely, we use reinforcement learning (RL) to train reasoning models to dynamically allocate inference-time compute based on task complexity. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy, thereby achieving substantial efficiency gains. It enables the derivation of a family of reasoning models with varying efficiency levels, controlled via a single hyperparameter. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。