用模型自信度做奖励，无需外部标注就能提升推理能力
Maximizing Confidence Alone Improves Reasoning
- 以模型输出的熵作为内在奖励，实现无监督强化学习
- 在GSM8K等5个推理基准上显著提升大模型表现
- 适合缺乏标注数据的场景，通用性强
强化学习（RL）已推动机器学习在多个领域取得进展，尤其使前沿语言模型具备解决复杂数学、科学与编程问题的能力。然而，强化学习的核心是奖励函数，而奖励设计历来极具挑战。本文提出RENT：基于熵最小化的强化学习方法——一种完全无监督的强化学习框架，无需外部奖励或真实答案，仅利用模型自身生成分布的熵作为内在奖励。实验表明，通过增强高置信度推理链，模型推理能力显著提升。我们在GSM8K、MATH500、AMC、AIME和GPQA等多个常用推理基准上验证了该方法的有效性，涵盖Qwen、Mistral和Llama系列不同规模的模型。该无监督学习方法具有广泛适用性，特别适用于缺乏外部监督的场景。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has enabled machine learning models to achieve significant advances in many fields. Most recently, RL has empowered frontier language models to solve challenging math, science, and coding problems. However, central to any RL algorithm is the reward function, and reward engineering is a notoriously difficult problem in any domain. In this paper, we propose RENT: Reinforcement Learning via Entropy Minimization -- a fully unsupervised RL method that requires no external reward or ground-truth answers, and instead uses the model's entropy of its underlying distribution as an intrinsic reward. We find that by reinforcing the chains of thought that yield high model confidence on its generated answers, the model improves its reasoning ability. In our experiments, we showcase these improvements on an extensive suite of commonly-used reasoning benchmarks, including GSM8K, MATH500, AMC, AIME, and GPQA, and models of varying sizes from the Qwen, Mistral, and Llama families. The generality of our unsupervised learning method lends itself to applicability in a wide range of domains where external supervision is unavailable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。