让大模型通过推理路径一致性自我奖励，无需人工标注即可提升逻辑能力。
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
- 用中间推理状态的一致性与波动性构建内在奖励信号
- 在多个推理任务上表现接近或超过有监督强化学习
- 适合无标注数据环境下提升大模型的自主推理能力
近期强化学习进展揭示了其在复杂推理任务中的潜力，但有效训练通常依赖外部监督，限制了广泛应用。本文提出一种新型自奖励强化学习框架，通过利用不同推理轨迹中中间状态的一致性来增强大语言模型的推理能力。核心洞察是：正确答案的中间推理状态往往在模型似然上趋于自身最终答案（高一致性），对其他候选答案偏离较小（低波动性）。受此启发，我们设计了CoVo机制，通过鲁棒的向量空间聚合策略整合一致性与波动性，并引入好奇心奖励以促进多样探索。CoVo使大模型能以自奖励方式开展强化学习，提供了一条无需外部监督的可扩展推理学习路径。在多个推理基准上的大量实验表明，CoVo性能达到甚至超越有监督强化学习水平。代码已开源：https://github.com/sastpg/CoVo。
原文摘要 · Abstract (English)
Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。