用验证机制筛选大模型推理答案,实现无需人工标注的强化学习。
JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR

- 先投票选候选答案,再用形式化验证判定是否给奖励。
- 在数学任务上达到与监督训练相当的准确率,且泛化能力更强。
- 适合追求低成本、高可靠推理的AI研究者和开发者。
基于可验证奖励的强化学习(RLVR)能提升大语言模型的推理能力,但传统方法依赖人工标注答案或精心设计的奖励规则。在可机器检验的领域,无标签替代方案如多数投票或大模型自评虽降低标注成本,却可能引入错误正例,导致训练不稳定。本文提出JURY-RL,一种无标签的RLVR框架,将答案生成与奖励判定分离:模型生成的回溯结果通过投票提出候选答案,再由形式化验证器(Lean)判断该答案是否可被认可。仅当多数票答案通过验证时,相关回溯才获奖励;若验证不确定,则启用ResZero作为备用奖励,对未验证的多数答案置零,并在剩余答案间保持梯度方差不变。在三个数学数据集上训练的骨干模型中,JURY-RL持续优于其他无标签基线,在数学推理、代码生成及通用基准上表现优异。其pass@1性能接近有监督训练水平,且在pass@k和响应多样性上展现更优泛化能力。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) enhances the reasoning of large language models (LLMs), but standard RLVR often depends on human-annotated answers or carefully curated reward specifications. In machine-checkable domains, label-free alternatives such as majority voting or LLM-as-a-judge remove annotation cost but can introduce false positives that destabilize training. We introduce JURY-RL, a label-free RLVR framework that decouples answer proposal from reward disposal: votes from model rollouts propose a candidate answer, and a formal verifier determines whether that candidate can receive positive reward. Concretely, only rollouts matching the plurality-voted answer are rewarded when that answer is successfully verified in Lean. When verification is inconclusive, we invoke ResZero (Residual-Zero), a fallback reward that discards the unverified plurality proposal and redistributes a zero-mean, variance-preserving signal over the residual answers. This design maintains a stable optimization gradient without reinforcing unverifiable consensus. Across three backbone models trained on mathematical data, JURY-RL consistently outperforms other label-free baselines on mathematical reasoning benchmarks and transfers competitively to code generation and general benchmarks. It attains pass@1 performance comparable to supervised ground-truth training, with superior generalization demonstrated by higher pass@k and response diversity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。