通过互教机制迭代优化策略与奖励模型，无需人工标注。
Mutual-Taught for Co-adapting Policy and Reward Models
- 用类似EM算法的两步循环：先用奖励模型指导策略更新，再用策略输出构建新奖励数据。
- 8B模型在AlpacaEval-2上达到54.1%的长度控制胜率，性能接近GPT-4o。
- 适合追求低成本强化学习优化、无标注场景下的模型改进研究者。
在大语言模型偏好优化过程中，新生成模型样本与训练奖励模型的数据之间可能产生分布偏移，导致奖励模型效能下降，进而影响策略模型表现。为此，我们提出互教（Mutual-Taught）方法，一种无需额外人工标注的自训练机制，可迭代优化策略模型（PM）与奖励模型（RM）。该方法模仿期望最大化（EM）算法：在E步中，利用当前奖励模型对策略模型进行反馈优化，使其更贴近潜在最优偏好分布；在M步中，基于策略模型更新前后的输出构造训练数据，用于更新奖励模型，确保其适应策略分布变化。实验表明，该迭代过程持续提升两模型性能。具体而言，我们的8B策略模型LLaMA-3-8B-Instruct-MT在AlpacaEval-2上实现54.1%的长度控制胜率，而8B奖励模型FsfairX-LLaMA3-RM-MT在RewardBench上的表现与GPT-4o-2024-08-06相当。
原文摘要 · Abstract (English)
During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, LLaMA-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1\% on AlpacaEval-2, while our 8B reward model, FsfairX-LLaMA3-RM-MT, performs on par with GPT-4o-2024-08-06 on RewardBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。