PPO在长思维链任务中失效，新方法通过校准价值函数解决根本问题。
What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- 预训练价值模型缓解初始化偏差
- 解耦GAE计算显著提升长序列奖励信号
- 适合需要长逻辑推理的LLM强化学习场景
强化学习（RL）对大语言模型（LLMs）生成复杂任务（如数学推理）所需的长思维链（CoT）至关重要。然而，尽管在多数场景有效，近端策略优化（PPO）在长CoT任务中表现失败。本文指出，价值初始化偏差和奖励信号衰减是导致失败的根本原因。为此提出价值校准的PPO（VC-PPO）：预先训练价值模型以消除初始化偏差，并将广义优势估计（GAE）计算解耦于智能体与评判器之间，以缓解奖励信号衰减。在美国邀请数学竞赛（AIME）上的实验表明，VC-PPO显著提升PPO性能。消融实验证明，这些技术对增强PPO在长CoT任务中的表现至关重要。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。