剖析离线偏好强化学习中关键设计对模型性能的影响
Design Considerations in Offline Preference-based RL
- 从理论角度分析损失函数、归一化策略与数据采样方式的作用
- 揭示不同方法在固定数据集下的性能差异机制
- 适合关注大模型对齐与算法设计的科研人员参考
基于人类偏好的离线强化学习（RLHF）方法，仅使用给定输入下生成响应的固定数据集及响应间的偏好反馈，已成为语言模型对齐研究的重要方向。本文从理论视角系统考察了DPO、IPO、SLiC等方法中的不同设计选择如何影响所学策略的质量，包括损失函数的设计、用于归一化对数似然的策略选取以及数据采样策略的作用。值得注意的是，我们的分析不依赖于传统再参数化论证，从而能统一处理该类方法的广泛变体。此外，我们在标准摘要任务基准上进行了小规模实证研究，验证了部分理论发现。
原文摘要 · Abstract (English)
Offline algorithms for Reinforcement Learning from Human Preferences (RLHF), which use only a fixed dataset of sampled responses given an input, and preference feedback among these responses, have gained increasing prominence in the literature on aligning language models. In this paper, we study how the different design choices made in methods such as DPO, IPO, SLiC and many variants influence the quality of the learned policy, from a theoretical perspective. Our treatment yields insights into the choices of loss function, the policy which is used to normalize log-likelihoods, and also the role of the data sampling policy. Notably, our results do not rely on the standard reparameterization-style arguments used to motivate some of the algorithms in this family, which allows us to give a unified treatment to a broad class of methods. We also conduct a small empirical study to verify some of the theoretical findings on a standard summarization benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。