arXiv:2507.15844cs.AIcs.CL2025-07被引 8

让大模型按问题难易自动调节推理深度,又快又准。

Hierarchical Budget Policy Optimization for Adaptive Reasoning

  • 用分层预算策略让模型学会根据问题复杂度自适应推理长度。
  • 在四个基准上平均节省60.6% token,准确率还提升3.14%。
  • 适合追求高效推理且不想牺牲模型能力的研究者和开发者。

大型推理模型通过大量思维链生成取得优异表现,但存在效率缺陷:对所有问题均采用统一的长推理过程。我们提出分层预算策略优化(HBPO),一种强化学习框架,使模型能在不损失能力的前提下学习特定问题的推理深度。与以往依赖固定约束或离散模式选择的方法不同,HBPO将探索空间划分为512–2560令牌的预算受限层级,各层级配备差异化奖励结构,同时保持效率激励与推理能力。该设计解决高效推理训练中的根本挑战:传统长度惩罚会系统性地引导模型避开必要长路径,导致探索空间坍缩。通过分层采样与预算感知奖励,HBPO维持探索多样性,教会模型识别何时需要深入推敲。大量实验表明,HBPO在四个推理基准上平均减少60.6%的令牌使用量,同时提升3.14%的准确率。尤为显著的是,模型展现出涌现的自适应行为,能根据问题复杂度自动调整推理深度。结果表明,推理效率与能力并非不可兼得,通过合理构建的分层训练可同时实现优化,并保持探索多样性。

原文摘要 · Abstract (English)

Large reasoning models achieve remarkable performance through extensive chain-of-thought generation, yet they suffer from a critical inefficiency: applying uniformly extensive reasoning regardless of problem complexity. We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability. Unlike existing approaches that impose rigid constraints or rely on discrete mode selection, HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities. This design addresses a fundamental challenge in efficient reasoning training: traditional length penalties systematically bias models away from necessary long reasoning paths, causing exploration space collapse. Through hierarchical sampling and budget-aware rewards, HBPO maintains exploration diversity while teaching models to recognize when extended deliberation is warranted. Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks. Most notably, HBPO exhibits emergent adaptive behavior where models automatically adjust reasoning depth based on problem complexity. Our results suggest that reasoning efficiency and capability are not inherently conflicting, and can be simultaneously optimized through appropriately structured hierarchical training that preserves exploration diversity.

推理优化强化学习自适应推理高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。