让大模型在推理出错时提前终止，节省计算并提升准确率
ESPO: Early-Stopping Proximal Policy Optimization
- 基于采样时的logits实时计算后悔值，动态判断是否失败并提前停止生成
- 在数学推理任务上显著超越PPO，AIME2024准确率提升至46.28%，节省超20%生成token
- 无需额外奖励模型或人工标注，自动聚焦错误位置的误差修正
当大语言模型在强化学习中早期产生错误推理时，传统算法仍强制其生成到最大长度，导致大量无正反馈的计算浪费，并污染优势估计。本文提出ESPO（Early-Stopping Proximal Policy Optimization），在采样过程中实时检测轨迹失败，提前终止生成。每步利用已计算的logits构造代理后悔值，当平滑累积后悔值显著高于预期时即终止。截断轨迹被视作吸收失败状态，赋予终态奖励，使负时间差（TD）误差集中于检测到的失败点，无需额外奖励模型或人工标注。在DeepSeek-R1-Distill-Qwen-7B模型上训练数学推理任务，ESPO在AIME~2024（46.28% vs. 45.25%）、AMC~2023（85.83% vs. 82.94%）和MATH-500（87.42% vs. 85.43%）上均优于PPO，累计节省超过20%的回滚生成token。
原文摘要 · Abstract (English)
When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure noise. We propose ESPO (Early-Stopping Proximal Policy Optimization), which detects trajectory failure on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the smoothed cumulative regret significantly exceeds its estimated values. Truncated trajectories are treated as absorbing failure states with a terminal reward, concentrating negative temporal-difference (TD) errors near the detected failure step without any additional reward model or human annotation. On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME~2024 (46.28% vs. 45.25%), AMC~2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%), while saving more than 20% rollout tokens cumulatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。