贪心采样在RLHF中效率更高，理论证明优于传统乐观策略。
Greedy Sampling Is Provably Efficient for RLHF
- 直接使用经验估计（贪心采样）实现最优性能
- 相比已有方法，理论误差阶数显著降低
- 适合关注强化学习理论与高效算法的研究者
从人类反馈中进行强化学习（RLHF）已成为大语言模型后训练的关键技术。尽管其在实践中表现优异，但理论理解仍有限，因仅凭偏好反馈学习带KL正则的目标带来额外挑战。现有工作多基于奖励型布拉德利-特瑞模型（BT），并沿用乐观或悲观设计。本文改而考虑更通用的偏好模型（近期实证中已显现其重要性），在理论上取得阶数级提升的性能保证。令人惊讶的是，这些结果来自直接使用经验估计的算法（即贪心采样），而非以往研究中的构造性乐观/悲观估计。这一发现源于在KL正则目标下最优策略类的独特结构特性，并进一步推广至BT模型，揭示了贪心采样在RLHF中的充分有效性。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use the empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。