arXiv:2605.23772cs.AIcs.LO2026-05被引 1

用智能体系统验证程序,成功率超九成,发现现有评测方法有漏洞。

Agentic Proving for Program Verification

论文配图:Agentic Proving for Program Verification
图 1 · 摘自论文原文
  • 让AI自动生成并验证代码规范与实现,全程自主推理。
  • 98.8%问题生成有效规范,87.5%正确验证实现,端到端成功率达98.1%。
  • 能精准指出自身错误和数据集中的隐藏缺陷,适合高阶验证研究者。

近年来,智能体系统在形式数学自动定理证明中表现卓越。为评估其在程序验证中的适用性,我们在CLEVER(基于Lean 4的可验证代码生成基准)上测试了Claude Code在智能体证明框架下的表现。结果表明,Claude对98.8%的问题生成了可接受的有效规范(其中81.3%通过了CLEVER基于同构性的评分),在正确基准上对87.5%的问题成功验证了实现,并在前提自洽的条目上实现了98.1%的端到端生成与验证成功率。在整个流程中,Claude还提供了高质量的自我反馈(经人工审查确认),能准确识别失败原因与数据集中残留的错误。这些发现揭示了现有程序验证基准难度与现代智能体证明能力之间的日益脱节,提示需采用更严格、抗错性强的评估方法,尤其应考虑替代同构性评分。更广泛地,实证显示,紧耦合编译器的智能体范式是当前最有效的基础程序验证路径。

原文摘要 · Abstract (English)

Agentic systems have recently emerged as state-of-the-art approaches for automated theorem proving in formal mathematics. To assess how far these capabilities extend to program verification, we evaluate Claude Code in an agentic proving framework on CLEVER, a Lean 4 benchmark for verifiable code generation. Our results show that Claude generates arguably valid specifications for 98.8% of problems (with 81.3% also accepted by CLEVER's isomorphism-based scoring on the correct portion of the benchmark), certifies implementations against correct ground-truth specifications for 87.5% of problems, and reaches a 98.1% success rate on the end-to-end program generation and verification pipeline over entries with self-consistent premises. Across all stages, Claude further provides high-quality feedback on its own attempts (as confirmed under manual review), identifying underlying causes of failure and lingering bugs in the dataset. These findings highlight a growing mismatch between the difficulty of existing program verification benchmarks and the capabilities of modern agentic provers, and point to the need for more rigorous, bug-resilient evaluation methodologies, and in particular for alternatives to isomorphism-based scoring of generated specifications. More broadly, our results provide empirical evidence that tight compiler-in-the-loop agentic paradigms are currently the most effective approach for foundational program verification.

程序验证智能体形式化方法AI推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。