让大模型自己重写推理过程,提升思考质量与效率。
Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
- 模型自重写推理文本,通过重写版本学习优化内部思考逻辑。
- 准确率提升0.6%,推理长度减少46%,无需额外提示控制长度。
- 有效缓解过度思考、冗余思考等问题,适合高要求推理场景。
通过基于结果正确性的强化学习,大规模推理模型在复杂任务中表现优异。但仅依赖最终正确性奖励,难以对内部推理过程提供细致监督,导致出现过度思考、不足思考、冗余思考和混乱思考等问题。受大模型自评分进展启发,本文提出自重写框架:模型重写自身推理文本,并从重写结果中学习以改进内部思考质量。算法上采用选择性重写策略,仅对模型一致正确的简单样本进行重写,保留原有GRPO的全部奖励信号。实现上将重写与原始生成合并至单一批次,保持强化学习可扩展性,仅增加约10%开销。在多种任务与不同规模模型上的实验表明,该方法在准确率-长度权衡上表现更优(准确率+0.6,推理长度-46%),且在LLM-as-a-judge评估下内部推理质量显著提升(+7.2分),有效缓解了内在推理缺陷。
原文摘要 · Abstract (English)
Through reinforcement learning (RL) with outcome correctness rewards, large reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal reasoning process. This deficiency leads to suboptimal internal reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own reasoning texts, and subsequently learns from the rewritten reasoning to improve the internal thought process quality. For algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL algorithm and introducing only ~10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter reasoning (-46%) even without explicit instructions in rewriting prompts to reduce reasoning length, outperforming existing strong baselines. In terms of internal reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal reasoning flaws.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。