用对比分歧法优化偏好，让奖励模型自动生成难样本。
Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator
- 将偏好优化视为负对数似然最小化，通过采样估计归一化常数。
- 提出的MC-PO在主流对齐基准上超越现有最优方法，性能提升显著。
- 适合研究大模型对齐、强化学习或需要高效负样本生成的场景。
现有的偏好优化研究主要依赖简单启发式方法构建成对偏好数据，例如基于人类或AI评分最大化优选与非优选结果之间的差距。然而这些启发式方法缺乏完整的理论依据。本文提出一种新的偏好优化框架，为有效采样非优选完成提供理论指导。我们将偏好优化建模为最小化概率模型的负对数似然（NLL），并提出通过采样策略来估计其归一化常数。我们证明，这些估计样本可作为偏好优化中的非优选完成。随后，选择对比分歧（CD）作为采样策略，提出一种新型蒙特卡洛偏好优化（MC-PO）算法，利用CD的蒙特卡洛核从参数化奖励模型中采样硬负样本。最后，提出在线版本的OnMC-PO算法。在多个主流对齐基准上，MC-PO优于现有最先进方法，而OnMC-PO进一步提升了性能。
原文摘要 · Abstract (English)
Existing studies on preference optimization (PO) have centered on constructing pairwise preference data following simple heuristics, such as maximizing the margin between preferred and dispreferred completions based on human (or AI) ranked scores. However, none of these heuristics has a full theoretical justification. In this work, we develop a novel PO framework that provides theoretical guidance to effectively sample dispreferred completions. To achieve this, we formulate PO as minimizing the negative log-likelihood (NLL) of a probability model and propose to estimate its normalization constant via a sampling strategy. As we will demonstrate, these estimative samples can act as dispreferred completions in PO. We then select contrastive divergence (CD) as the sampling strategy, and propose a novel MC-PO algorithm that applies the Monte Carlo (MC) kernel from CD to sample hard negatives w.r.t. the parameterized reward model. Finally, we propose the OnMC-PO algorithm, an extension of MC-PO to the online setting. On popular alignment benchmarks, MC-PO outperforms existing SOTA baselines, and OnMC-PO leads to further improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。