让大模型凭自我判断学习推理，无需外部奖励
Learning to Reason without External Rewards
- 用模型自身信心作为唯一奖励信号，实现无监督推理训练
- 数学题表现媲美有监督方法，代码生成泛化能力更强
- 适合无标注数据场景，推动自主智能系统发展
通过可验证奖励的强化学习（RLVR）训练大语言模型进行复杂推理虽有效，但受限于昂贵的领域特定标注。本文探索从内部反馈中学习的强化学习（RLIF）框架，使模型在无外部奖励或标注数据的情况下自主学习。提出Intuitor方法，以模型自身的置信度——自证度——作为唯一奖励信号，替代组相对策略优化（GRPO）中的外部奖励，实现完全无监督训练。实验表明，Intuitor在数学基准测试上达到与GRPO相当的性能，同时在代码生成等跨域任务中展现出更优泛化能力，且无需真解或测试用例。结果表明，内在模型信号可有效驱动跨领域学习，为不可获取可验证奖励的自主智能系统提供可扩展替代方案。代码已开源。
原文摘要 · Abstract (English)
Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor, an RLIF method that uses a model's own confidence-termed self-certainty-as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO's performance on mathematical benchmarks while achieving better generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。