提出新探索策略，让AI高效获取人类反馈，提升学习效率。
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
- 设计新采样方法，优先收集对优化策略最有帮助的反馈数据。
- 理论证明可实现多项式级后悔率，优于已有线性增长方法。
- 适合需要低样本成本、在线学习的智能体对齐场景。
基于人类反馈的强化学习（RLHF）已成为对齐大语言模型与人类偏好的核心范式，通过人类偏好数据训练奖励模型，并优化策略以生成更受青睐的响应。本文研究在线RLHF中的探索机制，目标是自适应地高效收集新偏好数据，以同时改进奖励模型与策略。分析现有基于乐观性的探索算法发现，其采样协议常选择无法有效降低奖励差异不确定性的比较，且理论证明此类方法在指数级长时域内可能产生线性后悔。为此，我们提出一种新探索方案，引导偏好查询聚焦于对策略改进最相关的奖励差异不确定性。在多臂老虎机模型下，我们建立了$T^{(β+1)/(β+2)}$阶的后悔上界，其中$β>0$为平衡奖励最大化与缓解分布偏移的超参数。据我们所知，这是首个在所有模型参数中后悔率呈多项式增长的在线RLHF算法。
原文摘要 · Abstract (English)
Reinforcement learning with human feedback (RLHF), which learns a reward model from human preference data and then optimizes a policy to favor preferred responses, has emerged as a central paradigm for aligning large language models (LLMs) with human preferences. In this paper, we investigate exploration principles for online RLHF, where one seeks to adaptively collect new preference data to refine both the reward model and the policy in a data-efficient manner. By examining existing optimism-based exploration algorithms, we identify a drawback in their sampling protocol: they tend to gather comparisons that fail to reduce the most informative uncertainties in reward differences, and we prove lower bounds showing that such methods can incur linear regret over exponentially long horizons. Motivated by this insight, we propose a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences most relevant to policy improvement. Under a multi-armed bandit model of RLHF, we establish regret bounds of order $T^{(β+1)/(β+2)}$, where $β>0$ is a hyperparameter that balances reward maximization against mitigating distribution shift. To our knowledge, this is the first online RLHF algorithm with regret scaling polynomially in all model parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。