将多轮对话对齐建模为马尔可夫博弈，提升语言模型与人类偏好一致性。
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees
- 将对齐问题转化为双人零和马尔可夫博弈，捕捉多轮交互动态。
- 理论证明需 𝒪(ε⁻¹) 次更新收敛至 ε-近似纳什均衡。
- 在多轮对话与数学推理数据集上验证有效性，优于传统方法。
基于人类反馈的强化学习（RLHF）在对齐大语言模型与人类偏好方面已取得显著成功。尽管现有方法如直接偏好优化（DPO）表现优异，但其将模型交互视为老虎机问题，难以适用于常见的多轮对话场景；且依赖布拉德利-特里模型假设，无法充分捕捉人类偏好的非传递性。本文提出将对齐问题建模为双人常和马尔可夫博弈，双方在对话所有步骤中均试图最大化对对方的胜率。所提方法乐观多步偏好优化（OMPO）基于乐观在线镜面下降算法。理论上，我们严格分析了OMPO的收敛性，证明其在 𝒪(ε⁻¹) 次策略更新内可收敛至 ε-近似纳什均衡。实验在多轮对话与数学推理数据集上验证了方法的有效性。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has been highly successful in aligning large language models with human preferences. While prevalent methods like DPO have demonstrated strong performance, they frame interactions with the language model as a bandit problem, which limits their applicability in real-world scenarios where multi-turn conversations are common. Additionally, DPO relies on the Bradley-Terry model assumption, which does not adequately capture the non-transitive nature of human preferences. In this paper, we address these challenges by modeling the alignment problem as a two-player constant-sum Markov game, where each player seeks to maximize their winning rate against the other across all steps of the conversation. Our approach Optimistic Multi-step Preference Optimization (OMPO) is built upon the optimistic online mirror descent algorithm~\citep{rakhlin2013online,joulani17a}. Theoretically, we provide a rigorous analysis for the convergence of OMPO and show that OMPO requires $\mathcal{O}(ε^{-1})$ policy updates to converge to an $ε$-approximate Nash equilibrium. We also validate the effectiveness of our method on multi-turn conversations dataset and math reasoning dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。