用重要性采样缓解大模型对齐中的过优化问题。
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
- 引入重要性采样比率，约束模型偏离参考策略
- 在低正则化条件下仍保持性能，优于现有方法
- 适合关注模型对齐稳定性与训练鲁棒性的研究者
直接对齐算法（DAAs）如直接偏好优化（DPO）已成为替代基于人类反馈强化学习（RLHF）的大型语言模型对齐方法。然而，这些方法更易出现过优化问题，即模型逐渐偏离参考策略，导致训练过程中性能下降。本文提出一种新的重要性采样方法（IS-DAAs），通过乘以考虑参考策略分布的重要性比率来缓解离线DAAs的过优化问题。IS-DAAs还通过截断重要性比率至最大值，避免了重要性采样带来的高方差问题。大量实验表明，该方法能有效缓解过优化，尤其在低正则化强度下表现更优，性能超越其他针对此问题设计的方法。代码已公开。
原文摘要 · Abstract (English)
Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel importance-sampling approach to mitigate the over-optimization problem of offline DAAs. This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our implementations are provided publicly at this link.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。