提升大模型强化学习的奖励信号精度，降低训练波动。
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
- 引入特权价值函数，精准注入任务相关的分步奖励信号
- 在多个推理任务中，优于标准价值函数基线
- 适合追求高稳定性和高精度的RL训练场景
大语言模型的强化学习算法主要区别在于方差减少策略。基于组相对的方法（如GRPO）通过每提示采样多个回溯序列来降低梯度方差，但仅提供序列级奖励，且受慢速回溯影响，导致吞吐量下降和策略偏离。学习到的价值函数理论上可解决上述问题，提供无需大批次采样的词元级优势。然而，额外的工程开销以及无评判器方法的实际成功，使得其在强化学习流程中难以推广。本文提出两种互补策略：1）特权价值函数（PVF），以无偏方式注入额外的任务相关词元级信号；2）TETHER，一种自适应基线，根据价值函数准确性动态插值于组相对与价值基线之间。在多个推理任务中，两种策略均一致优于标准价值函数基线，并达到或超越均值基线的GRPO表现。
原文摘要 · Abstract (English)
Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。