arXiv:2607.19407cs.AIcs.LO2026-07

首个跨形式化证明系统基准,评估大模型在不同系统间翻译定理的能力。

ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers

论文配图:ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers
图 1 · 摘自论文原文
  • 构建跨四类系统的统一验证框架,隔离基础逻辑差异。
  • 控制组证明翻译成功率29.7%,生态组仅5.2%,凸显库兼容性是主要瓶颈。
  • 适合研究大模型形式化推理、跨系统可移植性的学者使用。

形式化定理证明已成为机器学习的新前沿挑战,但生态系统碎片化:证明被锁在互不兼容的系统中,限制了基于学习的证明器训练数据,并阻碍已验证结果的可移植性。我们提出ITPEval,首个针对四大主流交互式定理证明器(Lean 4、Rocq、Isabelle、HOL Light)之间自动化定理翻译的基准,涵盖两种不同的逻辑基础。该基准包含1,560个源文件和6,848个定理,分为受控层级(隔离基础翻译难度)与生态层级(暴露API与证明风格差异)。我们发布itpeval,一个统一的多ITP验证基础设施,采用状态隔离的热后端,保持每个构件的原生检查语义。评估五种前沿开源大模型在12个定向翻译对上的命题与证明翻译表现:命题翻译最高通过率29.1% pass@1,证明翻译为10.5%;受控定理达29.7%证明通过率,而生态级翻译仅为5.2%,证实库不匹配是主要瓶颈。除pass@k外,通过确定性Lean 4 BEq检查发现,54.0%的验证通过的源到Lean 4 miniF2F命题翻译具备等价性,表明原生类型检查会高估语义保真度。在自动形式化/反形式化往返实验中,Rocq和HOL Light比Lean 4和Isabelle更易形式化,多ITP上下文将合并的Lean 4成功率从4.8%提升至10.6%。本研究的基准、验证基础设施与评估流程均已公开。

原文摘要 · Abstract (English)

Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results. We present ITPEval, the first benchmark for evaluating automated formal proof translation across four major ITPs (Lean 4, Rocq, Isabelle, and HOL Light), spanning two distinct logical foundations. Our benchmark comprises 1,560 source files and 6,848 theorems organized into a controlled tier of axiomatized files that isolates foundational translation difficulty, and an ecosystem tier drawn from real libraries that exposes API and proof-style mismatches. We release itpeval, a unified multi-ITP verification infrastructure with state-isolated warm backends that preserve per-artifact native checking semantics. We evaluate both statement and proof translation across five frontier and open-weight LLMs on 12 directed translation pairs: statement translation peaks at 29.1% pass@1 and proof translation at 10.5%; controlled theorems reach 29.7% proof pass@1 versus 5.2% for ecosystem-level translations, confirming that library mismatch is the dominant bottleneck. In addition to pass@k evaluation, a deterministic Lean 4 BEq check establishes equivalence for 54.0% of verified source-to-Lean 4 miniF2F statement translations, showing that native type-checking alone can substantially overestimate semantic fidelity; in an autoformalization/auto-informalization round-trip study, Rocq and HOL Light are easier formalization targets than Lean 4 and Isabelle, while multi-ITP context improves pooled Lean 4 success from 4.8% to 10.6%. Our benchmark, verification infrastructure, and evaluation pipelines are publicly released.

定理证明大模型跨系统基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。