首个形式化证明奖励模型评测基准,解决奖励信号稀疏问题。
FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models

- 构建250组偏好对,通过五种专家设计错误注入策略生成对比证明
- 前沿大模型最高准确率达59.8%,专用证明模型仅24.4%表现更差
- 揭示证明能力不等于评估能力,适合研究奖励模型与形式化数学的学者
近期神经定理证明器采用可验证奖励的强化学习(RLVR),由证明助手提供二值正确性信号。尽管该方法成本低且无奖励欺骗风险,但存在奖励稀疏问题:复杂问题中部分进展无法获得学习信号。为此,我们提出首个基于Lean 4的形式化定理证明奖励模型评测基准——FormalRewardBench。该基准包含250组偏好对,每对由正确证明与五种专家设计的错误变体配对生成:强制错误、单点微调、冗长错误证明、自然语言解释及Python代码注入。我们评估了前沿大模型(如Claude Opus 4.5)、判别类大模型(如CompassJudger-1-14B)、通用大模型(如Qwen2.5-72B-Instruct)和专用定理证明模型(如DeepSeek-Prover-V2-7B)。结果表明,前沿大模型表现最佳(59.8%),而专用模型最差(24.4%),说明定理证明能力无法迁移至证明质量评估。我们进一步分析各错误注入机制的挑战性,推动奖励模型在形式数学中的研究。
原文摘要 · Abstract (English)
Recent neural theorem provers use reinforcement learning with verifiable rewards (RLVR), where proof assistants provide binary correctness signals. While verifiable rewards are cheap and scalable without reward hacking issues, they suffer from sparse credit assignment: models receive no learning signal from difficult problems where partial progress goes unrewarded. This motivates learned reward models that can evaluate proof quality beyond binary verification. However, comparing reward models is challenging since it typically requires expensive RL training ablations. To address this, we introduce \textbf{FormalRewardBench}, the first benchmark for evaluating reward models in formal theorem proving with Lean 4. Our benchmark consists of 250 preference pairs where correct proofs are paired with incorrect variants generated through five expert curated error injection strategies: forced mistakes, minimal single-point variations, verbose incorrect proofs, natural language justification, and Python code injection. We evaluate frontier LLMs (e.g., Claude Opus 4.5), judge LLMs (e.g., CompassJudger-1-14B), general-purpose LLMs (e.g., Qwen2.5-72B-Instruct), and specialized theorem proving models (e.g., DeepSeek-Prover-V2-7B). Our results reveal that frontier LLMs achieve the highest performance (59.8\%) while specialized theorem provers perform the worst (24.4\%), suggesting that theorem proving ability does not transfer to proof evaluation. We provide further insights on various error injection mechanisms, highlighting the challenging nature of most injection mechanisms. We release \textbf{FormalRewardBench} publicly to encourage more research on developing reward models in formal mathematics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。