让大模型偏好更贴近人类，通过校准奖励值提升对齐效果
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
- 校准隐式奖励的尺度，使其与真实奖励可比
- 在多个基准上显著优于现有对齐方法
- 适合追求高精度偏好对齐的研究者和开发者
我们研究大语言模型（LLMs）与人类偏好数据对齐的问题。对比偏好优化在利用可用偏好数据对齐 LLMs 时表现良好，其通过优化策略相关的隐式奖励来实现。然而，对比目标主要关注两个响应的隐式奖励相对值，而忽略了它们的实际数值，导致与人类偏好的对齐效果不理想。为解决此问题，我们提出校准直接偏好优化（Cal-DPO），一种简单而有效的算法。我们证明，仅通过校准隐式奖励，使其学习到的奖励值在量级上与真实奖励可比，即可显著提升对齐效果。理论分析表明，Cal-DPO 优于现有方法。在多种标准基准上的实验结果表明，Cal-DPO 显著优于现成的对齐方法。
原文摘要 · Abstract (English)
We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。