让大模型学会根据问题难易动态分配推理资源，提升解题效率。
Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization
- 基于推理预算的效用最大化，训练模型识别问题难度并合理分配资源。
- 在2.16倍和4.32倍预算下，数学解题准确率比LLaMA3.1提升4.14%和5.74%。
- 适合需要高效推理的复杂任务场景，尤其适用于资源受限环境。
解决数学问题一直是大语言模型的一项引人注目的能力，许多研究通过延长推理链（如自我修正和长链思维）来提升表现。然而，这些复杂模型表现出单一模式行为：简单问题也需冗长推理链。本文提出一种新方法，通过将推理预算感知建模为带约束的效用最大化问题，命名为推理预算约束策略优化（IBPO）。经IBPO微调的模型能理解问题难度，并将推理资源优先分配给更难的问题。在不同推理预算下，最佳模型在MATH500数据集上的准确率分别较LLaMA3.1 8B Instruct提升4.14%和5.74%（相对提升8.08%和11.2%），且性能提升约为自一致性方法的两倍。
原文摘要 · Abstract (English)
Solving mathematics problems has been an intriguing capability of large language models, and many efforts have been made to improve reasoning by extending reasoning length, such as through self-correction and extensive long chain-of-thoughts. While promising in problem-solving, advanced long reasoning chain models exhibit an undesired single-modal behavior, where trivial questions require unnecessarily tedious long chains of thought. In this work, we propose a way to allow models to be aware of inference budgets by formulating it as utility maximization with respect to an inference budget constraint, hence naming our algorithm Inference Budget-Constrained Policy Optimization (IBPO). In a nutshell, models fine-tuned through IBPO learn to ``understand'' the difficulty of queries and allocate inference budgets to harder ones. With different inference budgets, our best models are able to have a $4.14$\% and $5.74$\% absolute improvement ($8.08$\% and $11.2$\% relative improvement) on MATH500 using $2.16$x and $4.32$x inference budgets respectively, relative to LLaMA3.1 8B Instruct. These improvements are approximately $2$x those of self-consistency under the same budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。