新算法让强化学习对齐模型更省样本，突破指数级扩展瓶颈。
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
- 通过偏好激励的自探索机制，避免传统方法的指数级样本开销。
- 理论证明其样本复杂度为多项式级，实测在多个场景下显著优于基线。
- 特别适合偏好极端不均衡的任务，如唯一正确答案的问题求解。
基于人类反馈的强化学习（RLHF）已成为大语言模型对齐的关键技术。本文研究在线RLHF设置下的样本效率问题。现有在线RLHF算法，无论采用被动或主动探索，其样本复杂度均随奖励函数规模呈指数增长，这一根本限制使得它们在偏好严重偏斜的场景（如仅有一个正确答案的问题）中表现不佳。为此，我们提出自探索偏好激励在线偏好优化（SE-POPO），首次实现样本复杂度关于奖励规模的多项式级增长，回应了Xie等（2024）提出的开放问题。理论上，我们证明SE-POPO的样本复杂度优于现有探索算法；实验上，系统评估表明，其在两大主流RLHF应用场景及公开基准上均显著优于探索与非探索基线，标志着RLHF算法设计的重要进展。代码已开源：https://github.com/MYC000801/SE-POPO。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment. This paper studies the setting of online RLHF and focus on improving sample efficiency. All existing algorithms in online RLHF, whether doing passive exploration or active exploration, suffer from a sample complexity that scales exponentially with the scale of the reward function. This fundamental limitation hinders their effectiveness in scenarios with heavily skewed preferences, e.g. questions with a unique correct solution. To address this, we introduce Self-Exploring Preference-Incentive Online Preference Optimization (SE-POPO), an online RLHF algorithm that for the first time achieves a sample complexity that scales polynomially with the reward scale, answering an open problem raised by Xie et al. (2024).. Theoretically, we demonstrate that the sample complexity of SE-POPO dominates that of existing exploration algorithms. Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design. The code is available at https://github.com/MYC000801/SE-POPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。