arXiv:2602.00927cs.LG2026-02

延长推理时间能提升模型泛化能力,即使性能已饱和。

Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision

  • 通过增加训练时推理步数,提升模型在未知数据上的表现。
  • 在分布内性能饱和后,分布外性能仍持续提升。
  • 适合关注模型鲁棒性与泛化能力的研究者。

让大模型在训练时进行更长时间的推理,已成为构建解决复杂问题前沿模型的关键。近期工作通过强化学习微调以激发长链思维或通过架构递归扩展隐式推理。本文首次发现:在仅基于结果监督的条件下,随着训练时推理长度(如强化学习中的令牌预算或循环变压器的循环次数)增加,即使分布内(ID)性能已饱和,分布外(OOD)性能仍可继续提升。这表明鲁棒性可能需要比仅依赖分布内验证更大的计算预算。理论分析揭示两种机制:(i) 自迭代可增强假设空间的归纳偏置,重塑最优解以提升分布外泛化;(ii) 当短路解在分布内有效但在分布外失效时,随自迭代次数增加,正则化可降低模型对这些短路解的依赖。实验验证了两种实现方式:在合成任务中增加循环变压器的循环次数,以及在数学推理任务中增加强化学习微调的令牌预算。

原文摘要 · Abstract (English)

Training LLMs to think and reason for longer has become a key ingredient in building state-of-the-art models that can solve complex problems previously out of reach. Recent efforts pursue this in different ways, such as RL fine-tuning to elicit long CoT or scaling latent reasoning through architectural recurrence. This makes reasoning length an important scaling knob. In this work, we identify a novel phenomenon (both theoretically and experimentally): under outcome-only supervision, out-of-distribution (OOD) performance can continue improving as training-time reasoning length (e.g., the token budget in RL, or the loop count in looped Transformers) increases, even after in-distribution (ID) performance has saturated. This suggests that robustness may require a larger budget than ID validation alone would indicate. We provide theoretical explanations via two mechanisms: (i) self-iteration can induce a stronger inductive bias in the hypothesis class, reshaping ID-optimal solutions in ways that improve OOD generalization; and (ii) when shortcut solutions that work for ID samples but not for OOD samples persist in the hypothesis class, regularization can reduce the learned solution's reliance on these shortcuts as the number of self-iterations increases. We complement the theory with empirical evidence from two realizations of scaling training-time reasoning length: increasing the number of loops in looped Transformers on a synthetic task, and increasing token budgets during RL fine-tuning of LLMs on mathematical reasoning.

推理长度泛化能力强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。