提出新模型分离偏好中的传递性与循环性，提升大模型对复杂人类偏好的对齐效果。
Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
- 将偏好分解为传递性（标量）和循环性（向量）两部分，显式解耦。
- 在RewardBench 2上比基线提升1.23%，在纠缠偏好场景收敛更快、准确率更高。
- 适合需要处理复杂、非严格偏好的大模型对齐任务，如对话系统优化。
标准强化学习人类反馈（RLHF）依赖传递性标量奖励，无法捕捉人类偏好的循环特性。尽管通用偏好模型（GPM）有所改进，但其隐式建模将层次性与循环性混杂，无法保证最优解。为此，本文提出混合奖励-循环（HRC）模型，利用博弈论分解，显式分离偏好为正交的传递性（标量）与循环性（向量）分量。同时引入动态自对弈偏好优化（DSPPO），将对齐视为时变博弈，逐步引导策略趋向纳什均衡。合成数据实验验证了HRC在混合传递-循环场景下的结构优势：收敛更快、准确率更高。在RewardBench 2上，HRC持续优于基线方法（如GPM、BT），例如在Gemma-2B-it上提升1.23%。尤其在“平局”领域表现优异，证实其对复杂非严格偏好的鲁棒性。下游评估在AlpacaEval 2.0、Arena-Hard-v0.1、MT-Bench上均有效，使用Gemma-2B-it作为基础偏好模型时，HRC+DSPPO在AlpacaEval 2.0上达到44.75%的峰值胜率，在Arena-Hard-v0.1上达46.8%，显著超越以BT或GPM训练的SPPO基线。代码已开源。
原文摘要 · Abstract (English)
Standard RLHF relies on transitive scalar rewards, failing to capture the cyclic nature of human preferences. While some approaches like the General Preference Model (GPM) address this, we identify a theoretical limitation: their implicit formulation entangles hierarchy with cyclicity, failing to guarantee dominant solutions. To address this, we propose the Hybrid Reward-Cyclic (HRC) model, which utilizes game-theoretic decomposition to explicitly disentangle preferences into orthogonal transitive (scalar) and cyclic (vector) components. Complementing this, we introduce Dynamic Self-Play Preference Optimization (DSPPO), which treats alignment as a time-varying game to progressively guide the policy toward the Nash equilibrium. Synthetic data experiments further validate HRC's structural superiority in mixed transitive--cyclic settings, where HRC converges faster and achieves higher accuracy than GPM. Experiments on RewardBench 2 demonstrate that HRC consistently improves over both BT and GPM baselines (e.g., +1.23% on Gemma-2B-it). In particular, its superior performance in the Ties domain empirically validates the model's robustness in handling complex, non-strict preferences. Extensive downstream evaluations on AlpacaEval 2.0, Arena-Hard-v0.1, and MT-Bench confirm the efficacy of our framework. Notably, when using Gemma-2B-it as the base preference model, HRC+DSPPO achieves a peak length-controlled win-rate of 44.75% on AlpacaEval 2.0 and 46.8% on Arena-Hard-v0.1, significantly outperforming SPPO baselines trained with BT or GPM. Our code is publicly available at https://github.com/lab-klc/Hybrid-Reward-Cyclic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。