arXiv:2608.13522cs.LGcs.AI2026-08

首个评估AI生成代码与证明协同能力的开源基准,测试其在真实多模块项目中的可信编程水平。

Vero: Can AI Agents Build Formally Verified Software Repositories?

论文配图:Vero: Can AI Agents Build Formally Verified Software Repositories?
图 1 · 摘自论文原文
  • 构建跨模块的联合代码与形式化证明生成评测框架
  • 43个真实项目实例中仅27个被完全解决,高难度项目无一达标
  • 支持错误检测机制,可验证规范不满足或参考代码错误

AI代理在编程中日益普及,但生成代码的正确性无法保证。形式化验证代码生成要求代理同时产出实现与机器可验证的规范证明,是提升AI生成软件可信度的更强路径。现有基准要么聚焦单函数,要么仅评估已提供实现的证明生成。当前仍不清楚代理能否在真实的多模块代码库中做出一致的实现与证明选择。为此,我们提出Vero,首个在仓库级别评估联合实现与证明合成的基准。Vero包含43个来自真实仓库的多模块实例,涵盖Python、Dafny、Verus和Coq,涉及加密协议到分布式系统等多样领域。每个实例为预定义API接口的多模块Lean 4仓库,配有手动整理的形式化规范与参考实现,支持仅证明与代码+证明两种评估模式。为提升可靠性,Vero引入审计机制,允许代理正式证明规范不可满足或参考代码错误,从而在整理阶段发现并修正潜在错误。我们使用具备Lean工具链访问权限的前沿编码代理进行评估,最强代理仅完成27/43个实例,且在最困难的仓库中未能关闭任何规范。Vero为衡量向仓库级可信软件合成迈进的进展提供了具体测试平台,当前代理仍显不足。我们已在https://github.com/sunblaze-ucb/vero发布基准、整理流程与评估工具。

原文摘要 · Abstract (English)

AI agents are increasingly used for programming, but do not provide any guarantee on the correctness of generated code. Verified code generation, in which an agent produces both an implementation and a machine-checked proof of its specification, offers a stronger path toward trustworthy AI-generated software. Existing benchmarks in this direction either focus on individual functions or only evaluate proof generation with provided implementations. It is still an open question whether agents can make coherent implementation and proof choices across real multi-module codebases. To bridge this gap, we introduce Vero, the first benchmark to evaluate joint implementation and proof synthesis at the repository level. Vero contains 43 multi-module instances sourced from real-world repositories spanning Python, Dafny, Verus, and Coq, and covering diverse domains from cryptographic protocols to distributed systems. Each instance consists of a multi-module Lean 4 repository with predetermined API interfaces, manually curated formal specifications, and reference implementations, supporting both proof-only and code-and-proof evaluation modes. To improve benchmark reliability, Vero also includes an audit mechanism where agents are allowed to formally prove unsatisfiability of provided specification or incorrectness of reference code, which surfaces and corrects latent code and specification errors during curation. We evaluate frontier coding-agent configurations with Lean toolchain access. The strongest agent fully solves only 27 of 43 instances and closes no specifications on the hardest repositories. Vero provides a concrete testbed for measuring progress toward repository-scale verified software synthesis, where current agents still fall short. We release the benchmark, curation pipeline, and evaluation harness at https://github.com/sunblaze-ucb/vero.

形式验证AI编程代码生成基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。