arXiv:2608.20256cs.AI2026-08

模型自适应决定推理深度,既省算力又提准确率。

Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation

论文配图:Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation
图 1 · 摘自论文原文
  • 用三种模式选择推理长度,由模型自主决策
  • 平均响应长度减少41%,准确率仅微降0.014
  • 无需重训练即可跨数据集生效,尤其适合简单题

通过强化学习训练的推理语言模型通常使用固定的词元预算,导致简单问题过度计算、难题计算不足。本文让模型在回应首词中自主选择三种模式: extsc{NoThink}(快速作答)、 extsc{Short}(简短推理)或 extsc{Long}(长篇推理)。该选择通过组相对策略优化(GRPO)学习,结合分段奖励与每模式硬性词元上限,确保模式区分。在1.5B参数蒸馏模型上训练于MATH数据集,三种模式自然分化,且 extsc{Short}比 extsc{Long}更准确,说明模型按难度排序问题。平均三种子实验下,模型在保留测试集MATH500准确率(0.782 vs. 基线0.796)的同时,将平均响应长度从4,796降至2,811词元(减少41%)。该策略还可迁移至GSM8K等其他基准,简单题节省高达76%词元,且在相似长度下表现优于基线。

原文摘要 · Abstract (English)

Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: \textsc{NoThink} (answer as quickly as possible), \textsc{Short} (brief reasoning), or \textsc{Long} (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on MATH, the three modes emerge without collapsing to a single choice, and the brief modes end up more accurate than \textsc{Long}, which shows that the router sorts problems by difficulty rather than at random. Averaged over three seeds, the resulting policy stays close to the base model's accuracy on the held-out MATH500 ($0.782$ vs.\ $0.796$) while cutting the mean response length from $4{,}796$ to $2{,}811$ tokens (a $41\%$ reduction). Interestingly, it also transfers to other benchmarks without retraining, with the largest savings where problems are easier, with for instance 76\% token reduction on GSM8K and at higher accuracy than the baselines at similar response length. In short, we build a reasoning model that adaptively chooses how much to reason for each problem.

自适应推理算力优化强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。