提出JEPO算法，让大模型强化学习在无法验证的答案上也能有效训练。
Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data
- 用链式思考作为隐变量，改进证据下界以适配不可验证数据
- 在可验证任务上效果等同传统强化学习，在不可验证任务上优于监督微调
- 适合长文本生成、数学证明等缺乏明确标准答案的场景
我们提出一种新算法JEPO（Jensen's Evidence Lower Bound Policy Optimization），用于将强化学习扩展至不可验证数据。以往大多数工作聚焦于可验证数据（如短答案匹配），但当答案为长文本（如数学证明）时，该假设不再成立。为在现有训练约束下实现对不可验证数据的规模化强化学习，JEPO引入詹森证据下界，将链式思考视为生成过程中的隐变量。实验表明：在可验证数据（数学题）上，JEPO表现与基于可验证奖励的强化学习相当；在半可验证数据（Numina）上，优于仅依赖部分数据的软匹配评估方法；在完全不可验证数据（Numina-proof）上，其似然评估结果优于监督微调及多个消融基线。
原文摘要 · Abstract (English)
We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's Evidence lower bound Policy Optimization). While most prior efforts on scaling RL for LLMs focus on verifiable data where ground truth answers are typically short-form and can be matched easily; we investigate the case where such assumptions are less valid (e.g., when answers are long-form such as mathematical proofs). To scale RL training to unverifiable data with contemporary training constraints, we propose JEPO. JEPO applies Jensen's evidence lower bound, a pragmatic simplification of the evidence lower bound which views chain-of-thought as a latent variable in the generative process. We show that on verifiable data (math), JEPO is as effective as RL with verifiable rewards; on semi-verifiable data (numina), JEPO improves on soft-match based evaluations compared to RL with verifiable rewards which can only leverage a subset of the data source; finally, on unverifiable data (numina-proof), JEPO outperforms SFT and a few ablation baselines on likelihood evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。