不依赖理想假设，用博弈论方法让大模型更懂复杂人类偏好。
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent
- 将对齐问题建模为双人博弈，摆脱传统奖励假设限制。
- 理论证明收敛速度达 O(T⁻¹)，快于现有方法的 O(T⁻¹/²)。
- 实测优于主流RLHF算法，在多个基准上表现更优。
基于人类反馈的强化学习（RLHF）在对齐大语言模型（LLMs）与人类偏好方面表现出色。现有方法多依赖布拉德利-特雷西（BT）模型假设，即每个提示-响应对存在一个真实奖励。然而，该假设在建模复杂人类偏好时可能过于严格。本文放弃BT模型假设，将对齐问题置于一般偏好框架下，建模为两人博弈。借鉴博弈学习理论，引入乐观在线镜像下降（OMD）以逼近纳什策略。理论上，证明该方法使对偶间隙达到O(T⁻¹)收敛速度，优于先前O(T⁻¹/²)的结果。更重要的是，实验实现表明，该方法在多个代表性基准上均超越当前最优的RLHF算法。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences. Many existing alignment approaches rely on the Bradley-Terry (BT) model assumption, which assumes the existence of a ground-truth reward for each prompt-response pair. However, this assumption can be overly restrictive when modeling complex human preferences. In this paper, we drop the BT model assumption and study LLM alignment under general preferences, formulated as a two-player game. Drawing on theoretical insights from learning in games, we integrate optimistic online mirror descent into our alignment framework to approximate the Nash policy. Theoretically, we demonstrate that our approach achieves an $O(T^{-1})$ bound on the duality gap, improving upon the previous $O(T^{-1/2})$ result. More importantly, we implement our method and show through experiments that it outperforms state-of-the-art RLHF algorithms across multiple representative benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。