arXiv:2604.19780cs.CL2026-04被引 1

让大模型按难易程度分配推理算力,避免浪费或不足。

Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMs

  • 根据题目难易动态调整推理步数,避免简单题耗时、难题不够用。
  • 在数学推理任务上,相同算力下准确率最高提升8.3%,平均少用34% token。
  • 适合需要高效推理的场景,如资源受限部署或实时问答系统。

通过扩展推理阶段提升大语言模型(LLMs)能力已成为主流方法。然而,现有方法在固定或均匀采样的token预算下优化推理,忽视了问题难度与分配算力之间的根本不匹配,导致简单问题过度思考、复杂问题思考不足,影响不同推理场景下的算力效率。本文提出预算自适应课程推理(BCAE)框架,包含三个协同组件:(1) 基于连续预算条件的统一策略,将预算作为输入信号,无需分离思考与摘要策略;(2) 课程感知的预算调度器,根据实时学习进度动态调整训练预算分布,从简单题向难题转移;(3) 截断感知的密集奖励机制,通过过程级验证实现中间推理步骤的精细奖励分配。此外引入预算条件优势估计(BCAE),以采样预算为条件调节优势基线,提升策略梯度稳定性。在数学推理基准(MATH、GSM8K、AIME、Minerva Math)上的实验表明,该方法在所有预算下均优于强基线,紧预算下准确率最高提升8.3%,相比无约束推理平均减少34%的token消耗。

原文摘要 · Abstract (English)

Scaling test-time compute via extended reasoning has become a key paradigm for improving the capabilities of large language models (LLMs). However, existing approaches optimize reasoning under fixed or uniformly sampled token budgets, ignoring the fundamental mismatch between problem difficulty and allocated compute. This leads to overthinking on easy problems and underthinking on hard ones, resulting in suboptimal token efficiency across diverse reasoning scenarios. In this paper, we propose Budget-Adaptive Curriculum Reasoning (BCAE), a unified framework that jointly optimizes reasoning quality and token efficiency through three synergistic components: (1) a \emph{budget-conditioned unified policy} that embeds the token budget as a continuous conditioning signal, eliminating the need for decoupled thinking and summarization strategies; (2) a \emph{curriculum-aware budget scheduler} that adaptively shifts the training budget distribution from easy to hard problems based on real-time learning progress; and (3) a \emph{truncation-aware dense reward} mechanism that provides fine-grained credit assignment at intermediate reasoning steps via process-level verification. We further introduce \emph{Budget-Conditioned Advantage Estimation} (BCAE), a novel variance reduction technique that conditions the advantage baseline on the sampled budget, yielding more stable policy gradients. Experiments on mathematical reasoning benchmarks (MATH, GSM8K, AIME, and Minerva Math) demonstrate that BACR consistently outperforms other strong baselines across all token budgets, achieving up to 8.3\% accuracy improvement under tight budgets while reducing average token consumption by 34\% compared to unconstrained reasoning.

大模型推理预算调度数学推理效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。