用大模型自身生成答案的期望来评估正确性，实现无需外部验证的通用强化学习推理。
Reinforcement Learning with Conditional Expectation Reward
- 以生成答案为条件，计算参考答案的期望概率作为奖励信号。
- 在数学与通用推理任务中均表现优异，优于传统规则验证方法。
- 无需人工设计验证规则，适合自由回答场景，适合构建通用推理系统。
基于可验证奖励的强化学习（RLVR）在提升大语言模型的推理能力方面表现出色，尤其在数学等可通过规则验证的领域。然而，依赖手工设计的、领域特定的验证规则严重限制了RLVR在自由回答场景中的应用，因为这些场景下答案多样性高，难以建立完整准确的规则。为此，我们提出条件期望奖励（CER），利用大语言模型自身作为隐式验证器，从而适用于一般领域且无需外部验证器或辅助模型。CER定义为在给定生成答案条件下，生成参考答案的期望概率。相比规则验证器提供的二值反馈，CER提供软性、分级的奖励信号，更适应答案正确性存在差异的任务。实验表明，CER在涵盖数学与通用领域的多种推理任务中均有效，证明其作为灵活通用验证机制的潜力。代码已开源：https://github.com/changyi7231/CER。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism. The code is available at https://github.com/changyi7231/CER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。