提出记忆保持机制,解决大模型训练中已解问题逐渐失效的问题。
Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

- 设计回顾机制,定期重放已解决任务以强化记忆
- 实验显示在20个基准上性能优于GRPO等基线方法
- 适用于多模态和不同算法的强化学习任务
基于可验证奖励的强化学习(RLVR)虽提升大语言模型能力,但常隐藏代价:训练过程中已解决的问题会悄然失效。本文将此现象定义为“正确集流转”,即解题能力获得与退化的耦合动态,使保留能力成为与获取能力同等重要的优化目标。通过理论与实证分析,我们提出“修复窗口原则”:恢复退化提示的成本随复习延迟急剧上升,存在低成本修复窗口,而标准RLVR流程未能利用。为此,我们提出 extbf{ ame{}},一种感知记忆的回顾机制,通过追踪已掌握提示并周期性重放,提醒模型过往解法。借助预轮次批替换策略, ame{}实现零额外推理开销。在涵盖图像-文本、视频及纯文本任务的20个基准上,使用Qwen3-VL与Qwen2.5-Math进行评估, ame{}持续优于GRPO、DAPO及回放基线,展现出跨模态与算法的稳健泛化能力。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves the ability of large language model, yet headline accuracy gains often conceal a hidden cost: previously solved problems quietly become unsolvable as training proceeds. We frame this phenomenon as \emph{correct-set turnover}, representing the coupled dynamics of solution acquisition and regression over the mastered set. Under this view, retention becomes an explicit optimization target alongside acquisition. We analytically and empirically establish the \emph{repair-window principle}: the cost of restoring a regressed prompt grows sharply with review delay, defining a low-cost window that standard RLVR pipelines fail to exploit. To address this, we propose \textbf{\method{}}, a retention-aware review mechanism that tracks mastered prompts and periodically reintroduces them to \textbf{remind} the model of previous solutions. By utilizing pre-rollout batch replacement, \method{} incurs zero additional rollout overhead. Evaluated across 20 benchmarks spanning image-text, video, and text-only tasks with Qwen3-VL and Qwen2.5-Math, \method{} consistently improves performance over GRPO, DAPO, and replay baselines, demonstrating robust generalizability across modalities and algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。