让过期数据变有用，突破强化学习的效率瓶颈
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
- 用二阶矩约束重要性权重，只剔除极端噪声，保留有效更新
- 高延迟下剪裁词元比例从1.22%降至0.06%，训练更稳定
- 支持256步以上过期数据，适配大模型推理优化场景
强化学习推动了大语言模型推理的进展，但多数算法依赖每轮更新都需新采样的在线策略训练，限制了效率与可扩展性。异步强化学习系统通过解耦采样与训练缓解此问题，但其性能受制于对采样数据严重过时的容忍度，现有方法在此情境下性能下降甚至崩溃。我们重新审视该挑战，发现存在‘繁荣前崩溃’现象：若合理利用，过期数据仍可提供与在线数据相当的信息。基于此，提出M2PO（二阶矩信任策略优化），通过约束重要性权重的二阶矩，在抑制极端异常值的同时保留有益更新。显著降低高过时情况下的剪裁词元比例（训练中从1.22%降至0.06%），精准屏蔽高方差样本，维持稳定优化。在六种模型（1.7B至32B）和八个基准上评估表明，即使数据延迟达256次模型更新，M2PO仍能实现稳定离线策略训练，并达到在线策略性能。
原文摘要 · Abstract (English)
Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a prosperity-before-collapse phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce M2PO (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable off-policy training even with data stale by at least 256 model updates and matches on-policy performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。