用最大似然替代传统强化学习，提升采样效率与模型性能。
Maximum Likelihood Reinforcement Learning
- 以最大似然为目标，设计可随计算量调节的样本优化目标。
- 在多任务测试中超越现有方法，测试时效率提升最高达20倍。
- 适合追求高采样效率与训练扩展性的算法研究者。
强化学习（RL）是仅能通过模型采样评估目标函数场景下的首选方法。当反馈为终态且二值时，模型隐式定义了正确轨迹的似然分布。最大似然本应是自然框架，但因不可导性改用RL作为折中。本文证明标准期望奖励RL仅为似然的一阶近似。为此提出最大似然强化学习（MaxRL），一种基于计算量的样本目标族，可在期望奖励RL与最大似然间插值。该目标仅需一行代码修改即可融入标准RL实现。在所有测试模型与任务中，MaxRL均帕累托占优；相较GRPO，测试时效率最高提升20倍；且在更多训练数据与计算资源下展现更优扩展性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling from the model. Our key observation is that when the feedback is terminal and binary, models implicitly induce a likelihood over correct rollouts. Maximum likelihood would be the natural framework in such settings, but RL is used instead as a workaround to the non-differentiability. We prove that the standard, expected-reward RL formulation is only a first-order approximation of the likelihood. To remedy this mismatch, we introduce Maximum Likelihood Reinforcement Learning (MaxRL), a compute-indexed family of sample-based objectives that interpolate between expected-reward RL and maximum likelihood as sampling compute is scaled. The resulting objective is a one-line change to standard RL implementations. MaxRL Pareto-dominates existing methods in all tested models and tasks, achieves up to $20\times$ gains in test-time scaling efficiency over GRPO, and scales more favorably with additional training data and compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。