对比离线与在线强化学习，发现在线方法在语言模型微调中表现更优。
Bridging Offline and Online Reinforcement Learning for LLMs
- 采用直接偏好优化与组奖励策略，实现从离线到在线的平滑过渡
- 在线与半在线方法性能相当，均显著优于传统离线方法
- 联合验证性与非验证性任务奖励，提升多任务泛化能力
本文研究在从离线、半在线到全在线三种训练模式下，强化学习方法对大语言模型微调的有效性，涵盖可验证的数学任务和不可验证的指令遵循任务。通过多个基准评估，系统比较了在线与半在线的直接偏好优化（DPO）和组奖励策略（Group Reward Policy Optimization），结果出人意料地发现两者性能与收敛速度相近，且均显著优于离线方法。我们详细分析了训练动态与超参数选择策略以获得最优效果。最后，实验表明同时使用可验证与不可验证任务的奖励进行多任务训练，能有效提升两类任务的整体表现。
原文摘要 · Abstract (English)
We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。