让大模型自己生成反馈数据，持续优化奖励模型。
Self-Evolved Reward Learning for LLMs
- 奖励模型通过自我生成新训练数据来迭代优化。
- 仅用少量人工标注数据，就能显著提升奖励模型性能。
- 适合希望降低人工标注依赖的LLM对齐研究者。
基于人类反馈的强化学习（RLHF）是使语言模型与人类偏好对齐的关键技术，在GPT-4、ChatGPT和Llama 2等对话模型的成功中发挥核心作用。然而，训练可靠奖励模型（RM）面临挑战，其依赖高质量的人工标注或先进AI系统提供的标签，成本高且可能引入偏差。随着语言模型能力提升，人工反馈对性能的进一步增强效果减弱。本文提出自进化奖励学习（SER），让奖励模型主动生成额外训练数据以自我迭代优化。我们在多个数据集（如HH-RLHF和UltraFeedback）上，使用Mistral和Llama 3等模型进行了广泛实验，对比多种基线方法。结果表明，即使仅有少量人工标注数据，基于自反馈的学习也能稳健提升奖励模型表现，从而有效增强大语言模型的能力。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences, playing a pivotal role in the success of conversational models like GPT-4, ChatGPT, and Llama 2. A core challenge in employing RLHF lies in training a reliable reward model (RM), which relies on high-quality labels typically provided by human experts or advanced AI system. These methods can be costly and may introduce biases that affect the language model's responses. As language models improve, human input may become less effective in further enhancing their performance. In this paper, we propose Self-Evolved Reward Learning (SER), a novel approach where the RM generates additional training data to iteratively improve itself. We conducted extensive experiments on multiple datasets such as HH-RLHF and UltraFeedback, using models like Mistral and Llama 3, and compare SER against various baselines. Our results demonstrate that even with limited human-annotated data, learning from self-feedback can robustly enhance RM performance, thereby boosting the capabilities of large language models (LLMs). Resources of this paper can be found at https://aka.ms/ser
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。