arXiv:2506.18110cs.LGcs.AI2025-06被引 18

通过动态揭示推理片段,让模型逐步学会复杂推理。

RL for Reasoning by Adaptively Revealing Rationales

  • 根据模型表现动态调整监督长度,实现逐级学习。
  • 在长序列依赖任务中,比纯强化学习更有效。
  • 适合需要渐进式训练的数学推理任务。

序列生成问题的输出空间呈组合爆炸,专家示范难以随序列长度扩展,而强化学习又面临奖励稀疏问题。介于完全监督与无监督强化学习之间的部分监督领域尚未被充分探索。本文提出自适应回溯(AdaBack)——一种针对每个样本的课程学习算法,动态揭示目标输出的部分前缀。监督长度根据模型历史奖励信号自适应调整,使模型能通过正确部分解逐步学习完整推理链。我们研究了介于监督微调(SFT)与强化学习(RL)之间的中间范式,认为每样本课程学习不仅是效率与通用性的权衡,更能解决存在长序列隐含依赖的任务,而传统SFT和RL均无法泛化。在具有潜在奇偶性约束的合成任务中,AdaBack可稳定求解原本不可行的问题。在三个数学推理基准(DeepScaleR、MATH、GSM8k)上,该方法使模型解决仅靠强化学习无法完成的问题,通过逐步暴露部分解获得新的推理能力。

原文摘要 · Abstract (English)

Learning in the combinatorially large output space of sequence generation problems is challenging as providing expert demonstrations scales poorly with sequence length, and RL struggles with sparse rewards. Between dense demonstrations in supervised training and no demonstrations in reinforcement learning lies an underexplored regime: partial supervision. We ask whether some classes of sequence learning problems become efficiently learnable by exploiting this gap. We address this by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals a partial prefix of the target output. The supervision length is adjusted dynamically for each sample based on the model's past reward signal, allowing it to incrementally learn to complete reasoning chains by conditioning on correct partial solutions. We investigate this intermediate regime between SFT and RL and argue that per-sample curriculum learning is more than a trade-off between efficiency and generality--it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize. Using a synthetic task with latent parity constraints, we show that AdaBack reliably solves problems that are otherwise intractable. On three mathematical reasoning benchmarks, DeepScaleR, MATH, and GSM8k, we find that AdaBack enables models to solve problems that RL alone cannot, acquiring new reasoning capabilities through incremental exposure to partial solutions.

强化学习推理生成课程学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。