提出SPO方法，让语言模型高效学习任意轨迹数据，突破传统强化学习的采样瓶颈。
Soft Policy Optimization: Online Off-Policy RL for Sequence Models
- 基于软强化学习框架，可在线/离线混合训练序列模型策略
- 在代码竞赛中优于PPO，pass@10提升显著且训练更快更省内存
- 无需价值网络，支持多样化的策略输出，适合需要泛化能力的场景
基于强化学习的语言模型后训练几乎全部采用在线策略方法（如PPO），无法利用早期训练、专家生成或探索采样产生的任意序列数据，导致样本效率低、探索困难且策略多样性下降。异步PPO实现还需频繁昂贵的模型传输，且通常依赖占用大量内存的价值函数模型。本文提出软策略优化（SPO），一种简单、可扩展且理论严谨的软强化学习方法，适用于序列模型策略，可从任意在线与离线轨迹中学习，无需独立价值模型。在代码竞赛任务上，SPO在pass@10指标上优于PPO，训练速度更快、内存消耗更低，能有效利用离线数据，具备更好稳定性，并学习到更丰富的（即更软）策略。
原文摘要 · Abstract (English)
RL-based post-training of language models is almost exclusively done using on-policy methods such as PPO. These methods cannot learn from arbitrary sequences such as those produced earlier in training, in earlier runs, by human experts or other policies, or by decoding and exploration methods. This results in severe sample inefficiency and exploration difficulties, as well as a potential loss of diversity in the policy responses. Moreover, asynchronous PPO implementations require frequent and costly model transfers, and typically use value models which require a large amount of memory. In this paper we introduce Soft Policy Optimization (SPO), a simple, scalable and principled Soft RL method for sequence model policies that can learn from arbitrary online and offline trajectories and does not require a separate value model. In experiments on code contests, we shows that SPO outperforms PPO on pass@10, is significantly faster and more memory efficient, is able to benefit from off-policy data, enjoys improved stability, and learns more diverse (i.e. soft) policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。