提出混合偏好优化方法，兼顾离线数据与在线探索，提升对齐效率。
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
- 融合离线偏好数据与在线探索，降低对数据分布的依赖
- 理论证明样本复杂度最优，比纯离线或纯在线方法更高效
- 适合需要高样本效率的模型对齐场景，如资源受限部署
当前大语言模型对齐主要依赖人类反馈强化学习（RLHF），通常使用大量离线偏好数据。然而，离线算法对数据分布有严格集中性要求，难以满足；而纯在线探索虽可规避此问题，但因需主动询问偏好且实时执行，成本较高。本文提出一种新型混合偏好优化（HPO）方法，通过放松离线探索的集中性条件，并显著提升在线部分的样本效率，实现离线与在线的协同。首次为基于偏好反馈的混合强化学习提供了可证明最优的理论边界，给出了策略优化的样本复杂度上界及匹配的下界。实验表明，该方法在样本效率上优于纯离线和纯在线方案。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。