RLoop通过迭代重初始化缓解强化学习过拟合，提升模型泛化能力。
RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization
- 用成功轨迹构建专家数据集，通过拒绝采样微调重初始化策略。
- 相比基础强化学习，平均准确率提升9%，pass@32提升超15%。
- 适合需要稳定推理和泛化能力的大模型训练场景。
尽管可验证奖励的强化学习（RLVR）在训练大型推理模型方面表现强大，但其训练动态存在关键挑战：强化学习过拟合，即模型获得训练奖励却丧失泛化能力。分析表明，这是由策略过度专业化和训练过程中生成的多样化解决方案的灾难性遗忘所致。标准优化过程丢弃了这一宝贵的时间步间策略多样性。为此，我们提出RLoop，一种基于迭代策略初始化的自提升框架。RLoop将标准训练流程转化为良性循环：首先利用强化学习从当前策略出发探索解空间，然后筛选成功轨迹生成专家数据集；该数据集通过拒绝采样微调（RFT）用于精炼初始策略，为下一轮迭代提供更优起点。此探索与利用的循环通过迭代重初始化，将瞬态策略变化转化为稳健性能提升。实验表明，RLoop有效缓解遗忘，显著提升泛化能力，相比基线强化学习，平均准确率提升9%，pass@32提升超过15%。
原文摘要 · Abstract (English)
While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for training large reasoning models, its training dynamics harbor a critical challenge: RL overfitting, where models gain training rewards but lose generalization. Our analysis reveals this is driven by policy over-specialization and catastrophic forgetting of diverse solutions generated during training. Standard optimization discards this valuable inter-step policy diversity. To address this, we introduce RLoop, a self-improving framework built on iterative policy initialization. RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset. This dataset is used via Rejection-sampling Fine-Tuning (RFT) to refine the initial policy, creating a superior starting point for the next iteration. This loop of exploration and exploitation via iterative re-initialization effectively converts transient policy variations into robust performance gains. Our experiments show RLoop mitigates forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。