用Lean证明器提供细粒度反馈,提升定理证明的强化学习效果。
Process-Verified Reinforcement Learning for Theorem Proving via Lean

- 利用Lean解析证明步骤,获取局部正确与首次失败位置的结构化反馈。
- 在MiniF2F和ProofNet上,战术级监督比仅结果反馈提升显著。
- 适合对形式化推理与强化学习结合感兴趣的学者或开发者。
虽然基于可验证奖励的强化学习通常依赖单一二值验证信号,但符号证明助手在形式推理中能提供丰富且精细的结构化反馈。这一过程与奖励之间的差距凸显了密集且可靠反馈的重要性。本文展示,Lean证明器本身可作为符号过程预言机,在训练中提供结果级与战术级的验证反馈。证明尝试被解析为战术序列,Lean的展开标记同时指出局部正确的步骤及最早失败点,生成根植于类型论的密集、验证基础的信用信号。我们将这些结构化奖励融入基于GRPO的强化学习目标,采用首次错误传播与首词信用方法,平衡结果与过程层面的优势。在STP-Lean和DeepSeek-Prover-V1.5上的实验表明,战术级监督在多数设置下优于仅结果反馈的基线,在MiniF2F和ProofNet等基准上实现性能提升。除实证收益外,本研究还揭示更广泛视角:符号证明助手不仅是评估时的验证者,更可在训练阶段充当过程级奖励预言机。这为结合语言模型可扩展性与符号验证可靠性的一致强化学习框架开辟道路。
原文摘要 · Abstract (English)
While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback. This gap between structured processes and unstructured rewards highlights the importance of feedback that is both dense and sound. In this work, we demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level and fine-grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages. Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of language models with the reliability of symbolic verification for formal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。