arXiv:2508.17196cs.LGcs.AI2025-08被引 29

让大模型按预算思考,精准控制推理长度。

BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens

  • 用特殊控制符实时提醒模型剩余计算预算。
  • 在数学推理任务中,不同预算下表现均优于基线。
  • 适合需要低延迟、省资源的实时应用场景。

近期大型语言模型通过增加推理时计算来提升推理能力,但此策略带来显著延迟与资源开销,限制了其在时间敏感或成本敏感场景的应用。本文提出BudgetThinker框架,使大模型具备预算感知推理能力,可精确控制思维过程长度。方法在推理过程中周期性插入特殊控制符,持续告知模型剩余的令牌预算。配合两阶段训练流程:先通过监督微调(SFT)让模型熟悉预算约束,再通过基于课程学习的强化学习(RL)阶段,使用长度感知奖励函数优化准确率与预算遵守度。实验表明,BudgetThinker在多个挑战性数学基准测试中,于不同推理预算下均显著优于强基线。该方法为高效、可控的模型推理提供了可扩展且有效的解决方案,使先进模型更适用于资源受限和实时环境部署。

原文摘要 · Abstract (English)

Recent advancements in Large Language Models (LLMs) have leveraged increased test-time computation to enhance reasoning capabilities, a strategy that, while effective, incurs significant latency and resource costs, limiting their applicability in real-world time-constrained or cost-sensitive scenarios. This paper introduces BudgetThinker, a novel framework designed to empower LLMs with budget-aware reasoning, enabling precise control over the length of their thought processes. We propose a methodology that periodically inserts special control tokens during inference to continuously inform the model of its remaining token budget. This approach is coupled with a comprehensive two-stage training pipeline, beginning with Supervised Fine-Tuning (SFT) to familiarize the model with budget constraints, followed by a curriculum-based Reinforcement Learning (RL) phase that utilizes a length-aware reward function to optimize for both accuracy and budget adherence. We demonstrate that BudgetThinker significantly surpasses strong baselines in maintaining performance across a variety of reasoning budgets on challenging mathematical benchmarks. Our method provides a scalable and effective solution for developing efficient and controllable LLM reasoning, making advanced models more practical for deployment in resource-constrained and real-time environments.

大模型推理预算控制强化学习高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。