让大模型先解题再自修正，提升推理与自我优化能力。
ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement
- 分两阶段联合优化模型解题与自修正能力，用同一奖励信号。
- 在Qwen3-4B上，自修正后通过率比基线高11.5个百分点。
- 训练中自动形成纠错优先、保正向强化的渐进式学习路径。
我们提出ThinkTwice，一种基于组相对策略优化（GRPO）的简单两阶段框架，联合优化大语言模型的推理与自我修正能力。每对训练步骤中，模型先解决推理问题，再对其自身解答进行修正，两个阶段均使用相同的二元正确性奖励，无需正确性信号或批评标注。在五个数学推理基准和两个模型家族（包括Qwen3-4B与Olmo3-7B）上，ThinkTwice显著优于现有在线策略优化基线。具体地，在Qwen3-4B上，未修正前其在AIME上的表现比GRPO高出5个百分点，经过一次自修正后提升至11.5个百分点（以pass@4衡量）。对训练动态的分析表明，ThinkTwice隐含一种‘先纠正、再强化’的课程机制：早期训练主要修正错误，随着模型性能提升，逐渐转向保留已正确答案，从而获得更精准的奖励信号。本工作确立了推理与自修正联合训练在强化学习验证与反思（RLVR）中的有效性和可解释性。
原文摘要 · Abstract (English)
We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。