改进直接偏好优化，让模型能处理无偏好的并列情况。
On Extending Direct Preference Optimization to Accommodate Ties
- 用广义布拉德利-特里模型替代原模型，显式建模并列关系。
- 实验显示引入并列数据后任务性能不降反升，且正则化更强。
- 适合需要精准偏好判断的场景，如机器翻译与数学推理。
本文推导并研究了两种显式建模并列关系的直接偏好优化（DPO）变体。通过将DPO中的布拉德利-特里模型替换为Rao-Kupper和Davidson提出的扩展模型，赋予并列以概率。在神经机器翻译和摘要任务上的实验表明，将明确标注的并列对加入数据集，不会导致性能下降，而传统DPO处理相同并列对时会出现性能退化。实证发现，包含并列项能增强相对于参考策略的正则化效果，即使在原始DPO中也可见此现象。我们基于理想DPO策略理论提供了该正则化效应的解释，并在翻译和数学推理任务上展示了所提变体优于原始DPO的表现。结果表明，在偏好优化中保留并列信息比简单丢弃更有效。
原文摘要 · Abstract (English)
We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。