用主动学习优化偏好数据收集，提升大模型训练效率
Active Learning for Direct Preference Optimization
- 线性化DPO目标，在网络末层计算最优反馈选择
- 反馈越多，偏好预测误差越小，理论可证
- 适用于在线/离线场景，适合大模型高效训练
直接偏好优化（DPO）是一种从人类反馈中学习的强化学习方法，策略直接由偏好反馈训练。尽管已有多种人类偏好模型，但如何选择最有效的反馈用于训练仍缺乏研究。本文提出一种面向DPO的主动学习框架，可用于在线收集人类反馈或离线筛选最具信息量的反馈子集。针对两种场景设计了高效算法：将DPO目标在策略网络最后一层线性化，并基于D-最优设计选择偏好反馈。理论上证明，随着反馈数量增加，DPO的对数几率估计误差持续减小。实验验证了该方法在符合理论假设的设置及大规模语言模型上的有效性。
原文摘要 · Abstract (English)
Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。