提出一种更鲁棒的对齐算法，降低人类反馈强化学习对模型假设的敏感性。
Doubly Robust Alignment for Large Language Models
- 采用双重稳健机制，只要偏好模型或参考策略之一正确就可保证性能
- 在多个基准上优于现有最优算法，且对错误假设更具容错能力
- 适合需要稳定微调的大模型对齐任务，尤其关注模型可靠性时
本文研究用于对齐大型语言模型与人类偏好的人类反馈强化学习（RLHF）。尽管RLHF已展现出良好效果，但许多算法对底层偏好模型（如Bradley-Terry模型）、参考策略或奖励函数的设定错误极为敏感，导致微调结果不佳。为缓解模型误设问题，本文提出一种双重稳健偏好优化算法，当偏好模型或参考策略其中之一正确时，该算法仍能保持一致性（无需两者同时正确）。所提方法在理论和实践中均表现优于现有最先进算法。代码已开源：https://github.com/DRPO4LLM/DRPO4LLM
原文摘要 · Abstract (English)
This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。