用少10倍数据实现与大模型离线训练相当的强化学习效果。
Efficient Exploration at Scale
- 边收集人类反馈边更新奖励模型和语言模型，实现在线学习。
- 仅用不到2万条标签就达到20万条标签的离线训练效果，数据效率提升超10倍。
- 适合需要快速迭代、减少标注成本的RLHF应用场景。
我们提出一种在线学习算法，显著提升基于人类反馈的强化学习（RLHF）的数据效率。该算法在接收选择数据时，增量式更新奖励模型和语言模型：奖励模型拟合选择数据，语言模型则通过一种变体的REINFORCE方法更新，其强化信号由奖励模型提供。关键设计包括：对每个强化信号添加微小正向激励、使用认知神经网络建模奖励不确定性、以及信息导向探索。在Gemma大型语言模型上，本算法仅需不足2万条标注即可达到离线RLHF在20万条标注下的性能，数据效率提升超过10倍；外推结果表明，训练100万条数据可媲美离线训练10亿条数据的效果，效率提升达1000倍。这是首次证明如此巨大提升的可行性。
原文摘要 · Abstract (English)
We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。