强化学习微调能用更少数据实现更好效果，关键在于先学简单验证器再优化生成器。
All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
- 先用偏好数据训练简单验证器，再通过强化学习优化生成器
- 相比离线最大似然，所需数据量显著减少
- 适合存在生成-验证差距的复杂生成任务
从基础原理看，基础模型微调中最强性能往往来自复杂的两阶段训练：先在偏好数据上训练奖励模型（RM），再用其提供在线反馈进行强化学习（RL）微调，而非直接对策略参数进行离线最大似然估计。事实上，信息论表明，经过奖励模型会丢失信息，而在线采样无法创造新信息。为解释这一矛盾，我们从理论与实证角度检验多个假设。结果发现，最合理的解释是：在存在生成-验证差距的任务中，（1）从偏好数据中学习简单的奖励模型（验证器）相对容易；（2）下游强化学习仅搜索对这类简单验证器最优的策略空间。因此，端到端两阶段在线微调只需在策略空间的子集内搜索，所需数据远少于离线微调。
原文摘要 · Abstract (English)
From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。