让大模型自我反思纠错，用内部反馈提升长序列推理能力
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
- 模型自检完成的推理过程，生成纠错提示
- 仅用8%训练算力达AIME'24 73.3%准确率
- 无需外部奖励模型，适合资源受限的长任务推理
自我反思是人类学习中重要的信用分配机制，能将稀疏的结果反馈转化为可操作的指导。然而其在后训练大语言模型中的潜力尚未充分探索。我们提出自反思策略优化（SRPO），使大模型具备内化该能力。SRPO使模型能够分析自身完成的推理轨迹，将错误归纳为简洁的“反思补丁”，并利用反思条件下的教师评分作为密集的分词级训练信号，在无需外部评论者、独立奖励模型或更大教师模型的情况下，将稀疏的终端监督转化为密集的分词级学习信号。我们在数学推理和长时程代理基准测试中均实现最先进性能，且数据效率极高。使用Qwen3-8B基础模型，仅需原始监督微调所需8%（0.08x）的训练浮点运算量，即在AIME'24上达到73.3%准确率，同时显著提升WebShop成功率（64.7%）、ALFWorld（76.8%）和SWE-Bench-Lite（31.2%）。代码已公开于https://github.com/Galleons2029/SRPO。
原文摘要 · Abstract (English)
Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。