让AI通过多次尝试纠错，提升复杂问题求解成功率。
Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought
- 设计可校准的强化学习策略，动态加权每轮尝试结果。
- 在5次尝试内成功解题率提升18.6%，优于基线方法。
- 适合需要高可靠性推理的复杂任务场景。
当前顶尖推理模型采用长链式思维（CoT）解决日益复杂的任务，依赖更多测试时计算。本文研究一种允许模型最多进行K次连续尝试的设定，每次尝试可基于前序尝试及硬验证反馈进行改进。为此，我们探索利用强化学习优化验证@K奖励（即模型在第K次尝试前成功）。发现直接按通过/失败结果加权各次尝试会导致梯度偏差。为此提出校准尝试级（CAL）GRPO方法，设计新权重策略以获得无偏梯度且保持低方差。理论揭示了逐次尝试奖励如何影响训练过程与最终验证@K性能。在合成数据与真实数据上的实验、基线对比与消融研究验证了理论正确性，证明CAL-GRPO在多项指标上优于原始GRPO与朴素加权方法。
原文摘要 · Abstract (English)
State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。