针对大模型对齐中的反馈偏差问题，提出更鲁棒的强化学习算法。
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
- 基于改进的奖赏模型，降低人类偏好判断误差的影响
- 在Anthropic数据集上77%-81%的输出优于基线方法
- 适合需要稳定对齐效果的模型微调场景
从人类反馈中进行强化学习（RLHF）已成为对齐大语言模型输出与人类偏好的关键技术。现有大多数RLHF算法使用布拉德利-特瑞模型来学习奖赏函数，该模型依赖于人类偏好假设，可能无法反映真实判断的复杂性和多样性。本文提出一种鲁棒算法，在奖赏模型误设情况下仍能提升性能。理论上，该算法降低了奖赏与策略估计器的方差，从而改善了后悔边界。在大语言模型基准数据集上的实证评估表明，所提算法持续优于现有方法，在Anthropic Helpful and Harmless数据集上77%-81%的响应被用户更偏好。代码已开源：https://github.com/VRPO/VRPO。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) has emerged as a key technique for aligning the output of large language models (LLMs) with human preferences. To learn the reward function, most existing RLHF algorithms use the Bradley-Terry model, which relies on assumptions about human preferences that may not reflect the complexity and variability of real-world judgments. In this paper, we propose a robust algorithm to enhance the performance of existing approaches under such reward model misspecifications. Theoretically, our algorithm reduces the variance of reward and policy estimators, leading to improved regret bounds. Empirical evaluations on LLM benchmark datasets demonstrate that the proposed algorithm consistently outperforms existing methods, with 77-81% of responses being favored over baselines on the Anthropic Helpful and Harmless dataset. The code is available at https://github.com/VRPO/VRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。