用参考模型生成的响应改进偏好优化，提升大模型输出质量。
Boosting Direct Preference Optimization with Penalization
- 在DPO基础上引入参考模型贪婪响应的惩罚项，仅当当前策略偏好错误时激活。
- 在AlpacaEval 2.0上，相比DPO等基线，胜率提升5.3%（Llama-3-8b-it）和4.4%（Gemma-2-9b-it）。
- 适合追求高效、无需强化学习的指令微调场景，尤其关注输出长度控制。
离线偏好优化已成为人类反馈强化学习的实用替代方案，但如直接偏好优化（DPO）及其变体的成对目标仅利用静态数据集中选定与拒绝的响应，忽略了参考模型对同一提示生成的响应所携带的有用信号。本文提出带惩罚的直接偏好优化（DPOP），在基础偏好损失上加入一个门控惩罚项，用于抑制参考模型贪婪生成的响应。该惩罚仅在当前策略对优选响应的似然仍低于拒绝响应时激活。在AlpacaEval 2.0测试中，DPOP在Llama-3-8b-it和Gemma-2-9b-it上均优于DPO、SimPO和AlphaDPO，实现相对5.3%和4.4%的胜率提升。消融实验进一步表明，类似SimNPO的长度归一化惩罚在此设置下强于NPO和词元级反似然。
原文摘要 · Abstract (English)
Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3\% and 4.4\% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。