通过隐式奖励边界提升大模型对齐效果，解决偏好数据过时问题
SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
- 利用隐式奖励边界筛选样本，减少对虚假特征的过拟合
- 用更新后的模型自举生成新偏好数据，提升对齐质量
- 可无缝集成现有方法，适合训练指令跟随类大模型
直接对齐算法（DAAs）如直接偏好优化（DPO）因简洁、高效和稳定，成为强化学习人类反馈（RLHF）的流行替代方案。然而，这些方法使用的偏好数据通常在对齐训练前收集且保持不变（离线策略），导致两个问题：(1) 策略模型可能捕捉数据集中的虚假相关性而非人类意图；(2) 过度拟合于更新后模型难以生成的离线轨迹反馈。为此，我们提出自检与对齐（SeRA），一种低成本高效的通用方法，可与现有DAAs结合使用。SeRA包含两部分：(1) 基于隐式奖励边界的样本选择，缓解对不良特征的过拟合；(2) 利用隐式奖励进行偏好自举，以低成本方式用更新的策略模型扩充偏好数据。大量实验，包括指令跟随任务，验证了SeRA在基于离线偏好数据训练大语言模型时的有效性和普适性。
原文摘要 · Abstract (English)
Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment training begins and remain unchanged (off-policy). This can lead to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by an updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。