提出稳定训练批评者的新方法，单次生成即可提升大模型推理性能。
Best Practice Critic Optimization
- 整合多种优化技术，构建稳定且精准的批评者训练方案。
- 在数学推理任务中，单次采样效果媲美多响应对比方法。
- 支持基于评分标准的奖励学习，适合需要可靠反馈的场景。
基于分组的强化学习方法（如GRPO）通过多次采样响应避免训练批评者，但可靠批评者可从单个响应中估算逐标记优势。然而，传统批评者训练常不稳定。本文研究该不稳定性，提出「最佳实践批评者优化」（BPCO），结合DPPO、奖励范围约束的价值预测、蒙特卡洛价值目标、未归一化策略优势及长度自适应广义优势估计。由于批评者仅用于训练，可引入策略不可见的奖励定义信息（如参考答案或评分标准）。控制实验验证各设计影响。在1.5B至30B-A3B混合专家模型的数学推理任务中，BPCO持续优于强基线，并在单次采样下达到或超越分组基线。相同方案亦提升基于评分标准的奖励学习效果。结果表明，精心设计的批评者可作为分组相对优势估计的可靠替代。代码已公开于https://github.com/QPHutu/golden_critic。
原文摘要 · Abstract (English)
Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best Practice Critic Optimization (BPCO)**, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。