解释了为何无价值函数强化学习在大模型中有效，揭示其更新本质为价值梯度。
Value-Gradient Hypothesis of RL for LLMs
- 从理论上证明无价值函数方法的更新等价于价值梯度
- 发现注意力机制自动微分可生成近似价值信号，误差可控
- 提出评估强化学习效果的两个关键指标，指导何时用最有效
强化学习显著提升预训练语言模型性能，但为何无需价值函数的方法（如PPO、GRPO）仍表现良好，以及何时收益最大，仍未被充分理解。本文从可微轨迹与加性噪声参数化出发，证明演员更新在期望下具有价值梯度特性：反向传播的协态变量的条件期望等于价值梯度。对于离散Transformer策略，我们发现通过注意力机制的自动微分能产生近似该价值信号的协态，其误差由采样间隔和策略熵控制。这些结果推动将强化学习影响分解为价值梯度信号与可实现奖励空间，从而给出沿预训练路径使用强化学习最有效的判断准则。
原文摘要 · Abstract (English)
Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains. We develop a value-gradient perspective of critic-free RL for LLM post-training. First, under a differentiable rollout and additive-noise parameterization, we show that the actor update is value-gradient-like in expectation: the backward pass propagates costates whose conditional expectation equals the value gradient. Second, for discrete transformer policies, we show that autodifferentiation through attention produces empirical costates that approximate this value signal, with an error controlled by the sampling gap and policy entropy. These results motivate a decomposition of RL impact into value gradient signal and reachable reward headroom, yielding a criterion for when RL should be most effective along a pretraining trajectory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。