用监督学习隐式耦合智能体与评论家，提升大模型推理训练稳定性。
Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
- 将奖励信号视为可预测标签，转为监督学习任务优化策略。
- 4B和8B模型分别提升8.26%和9.57%，超越主流基线。
- 适合需要稳定后训练的大型语言模型推理任务。
近期基于可验证奖励的强化学习（RLVR）进展使大语言模型（LLMs）能够处理数学与编程等复杂推理任务，但现有方法常面临稀疏奖励信号与强化学习固有的不稳定策略梯度更新问题。为此，我们提出PACS框架，通过监督学习实现智能体与评论家的隐式耦合。将结果奖励视为可预测标签，将RLVR问题重构为由策略模型参数化的评分函数的监督学习任务，并使用交叉熵损失进行优化。详细的梯度分析表明，该监督形式在本质上恢复了经典策略梯度更新，同时提供更稳定高效的训练。大量实验表明，PACS显著优于强开源模型与RLVR基线，在4B和8B模型上分别相对于基线提升+8.26%和+9.57%，为大语言模型后训练提供了有前景的新路径。代码与数据已开源：https://github.com/ritzz-ai/PACS。
原文摘要 · Abstract (English)
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming, however existing RLVR methods often suffer from sparse reward signals and unstable policy gradient updates inherent to RL-based approaches. To address the challenges, we propose PACS, a novel RLVR framework that achieves imPlicit Actor Critic coupling via a Supervised learning framework. By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. A detailed gradient analysis shows that this supervised formulation inherently recovers the classical policy gradient update while providing more stable and efficient training. Extensive experiments demonstrate that PACS significantly outperforms strong open-source models and RLVR baselines, yielding substantial average gains of +8.26% (4B) and +9.57% (8B) over base models offering a promising avenue for LLMs post-training with verifiable rewards. Our code and data are available as open source at https://github.com/ritzz-ai/PACS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。