arXiv:2410.19940cs.LOcs.AI2024-10被引 10

用大模型分而治之自动证明代码,还能结合人工提示提升成功率。

Cobblestone: A Divide-and-Conquer Approach for Automating Formal Verification

  • 大模型生成证明并拆解问题,逐个击破剩余难点。
  • 无需人工干预时,能证明超50%的定理,平均耗时14.7分钟。
  • 支持外部输入,配合提示可将成功率提升至58%,适合验证工具开发者。

使用证明助手(如Coq)进行形式化验证能有效提升软件质量,但需大量人力与专业知识。机器学习可自动合成证明,但仅能覆盖部分目标性质。我们提出Cobblestone,一种分而治之的证明合成方法。Cobblestone利用大语言模型(LLM)生成潜在证明,将其分解为更简单的子问题,自动识别已成功证明的部分,并对剩余部分迭代求解,最终构建出保证正确的证明,尽管依赖不完全可靠的LLM。我们在四个开源Coq项目基准上评估Cobblestone,控制训练数据泄露。在无须人工干预的情况下,其性能超越现有非LLM工具,且能证明其他基于LLM的工具无法解决的多个定理;在多个基准上表现更优。单次运行平均成本1.25美元,耗时14.7分钟。该方法亦支持外部输入(如用户或其它工具提供的证明结构或相关引理),在引入外部提示的条件下,可证明高达58%的定理。研究表明,结合部分进展与外部信息,可更高效地自动化形式化验证。

原文摘要 · Abstract (English)

Formal verification using proof assistants, such as Coq, is an effective way of improving software quality, but requires significant effort and expertise. Machine learning can automatically synthesize proofs, but such tools are able to prove only a fraction of desired software properties. We introduce Cobblestone, a divide-and-conquer approach for proof synthesis. Cobblestone uses a large language model (LLM) to generate potential proofs, uses those proofs to break the problem into simpler parts, automatically identifies which of those parts were successfully proven, and iterates on the remaining parts to build a correct proof that is guaranteed to be sound, despite the reliance on unsound LLMs. We evaluate Cobblestone on four benchmarks of open-source Coq projects, controlling for training data leakage. Fully automatically, Cobblestone outperforms state-of-the-art non-LLM tools, and proves many theorems that other LLM-based tools cannot, and on many benchmarks, outperforms them. Each Cobblestone run costs only $1.25 and takes 14.7 minutes, on average. Cobblestone can also be used with external input, from a user or another tool, providing a proof structure or relevant lemmas. Evaluated with such an oracle, Cobblestone proves up to 58% of theorems. Overall, our research shows that tools can make use of partial progress and external input to more effectively automate formal verification.

形式化验证大模型Coq分而治之

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。