让大模型在计算预算内快速生成越来越好的推理结果
Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data
- 用自生成偏好数据让模型在推理中自我优化
- 在有限token下，解的质量随推理量提升更快
- 适合需要实时响应的规划类任务
我们研究大语言模型在计算资源受限下的推理行为。在实际应用中，如行程规划，往往更看重在固定推理预算内尽快产出高质量中间结果，而非耗时的穷尽推理。为此，我们提出一种随时推理框架及“随时指数”（Anytime Index），量化解质量随推理令牌数增加的提升效率。进一步，我们设计一种推理时自改进方法，利用大模型自身推理过程的对比生成偏好数据，从而优化中间输出。在NaturalPlan（Trip）、AIME和GPQA数据集上的实验表明，该方法在Grok-3、GPT-oss、GPT-4.1/4o和LLaMA系列模型上均有效，显著提升了受限预算下的推理质量和效率。
原文摘要 · Abstract (English)
We study the reasoning behavior of large language models (LLMs) under limited computation budgets. In such settings, producing useful partial solutions quickly is often more practical than exhaustive reasoning, which incurs high inference costs. Many real-world tasks, such as trip planning, require models to deliver the best possible output within a fixed reasoning budget. We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase. To further enhance efficiency, we propose an inference-time self-improvement method using LLM-synthesized preference data, where models learn from their own reasoning comparisons to produce better intermediate solutions. Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models, improving both reasoning quality and efficiency under budget constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。