让大模型学会主动重试，提升推理准确率
$\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
- 通过强化学习教会模型在错误路径时主动放弃并重来
- 将重试行为从0.5%提升至30%以上，显著减少无效思考
- 适合需要高可靠推理的场景，如数学与逻辑任务
基于可验证奖励的强化学习（RLVR）通过增加测试阶段计算量，提升了大语言模型（LLM）的推理能力。然而，即使经过大量训练，这些模型仍会生成冗余且低质量的思维链步骤，导致效率低下和答案质量下降。我们发现，当初始思维链方向或质量不佳时，模型即使生成更多文本也难以得出正确答案。为此，我们提出强化学习重解机制（Re²），使模型学会在无效路径时灵活放弃并重启求解过程，而非强行给出最终答案。Re²采用纯强化学习，无需预训练微调，成功将原始模型中罕见的重试行为从0.5%提升至30%以上。在相同训练计算预算下，性能显著优于标准RLVR，且随着采样次数增加，测试阶段表现持续提升。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re$^2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re$^2$ applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。