提出持续推理训练框架，解决多任务强化学习中新增任务时模型性能下降问题。
Continual Reasoning Gym: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
- 通过持续更新已有模型应对新任务，避免重复训练。
- 发现模型存在轻微遗忘，但最终表现仍低于联合训练水平。
- 提出提示回放机制，利用共性推理结构提升长期性能。
基于可验证奖励的强化学习（RLVR）通常在多个任务上后训练推理模型，而随着新任务加入，重新运行多任务强化学习（MTRL）成本高昂。为此，我们研究持续强化学习中的推理能力扩展（continual RLVR），即在每个新任务到来时更新现有模型。核心问题是：持续更新的模型能否达到联合训练模型的表现？为此，我们构建了连续推理训练环境（Continual Reasoning Gym），将文本与视觉推理任务组织为五个任务序列。实验发现，顺序学习虽有轻微遗忘，但最终表现仍低于MTRL。进一步分解性能发现，遗忘仅解释部分差距；我们识别出共享推理现象——可迁移的推理结构使单个任务训练能平均支持其他任务。为此，我们提出持续提示回放（CPR）方法，通过回放旧任务提示并用当前策略重生成响应，以利用共享推理结构提升对当前及未来任务的学习效果。平均而言，仅有CPR达到MTRL级别的性能。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。