用AI自动生成Dafny程序的逻辑注解,提升代码验证成功率。
dafny-annotator: AI-Assisted Verification of Dafny Programs
- 结合大模型与搜索算法,自动为Dafny方法添加逻辑注解
- 在测试集上成功率从15.7%提升至50.6%,显著提高验证效率
- 适合想降低形式化验证门槛的开发者和研究者
形式化验证虽能大幅减少软件缺陷,但高昂的成本阻碍了其大规模应用。尽管Dafny有望显著降低编写可验证程序的难度,用户仍需提供逻辑注解以辅助验证器。本文探索利用大语言模型与搜索技术构建dafny-annotator:一个自动为Dafny方法添加逻辑注解直至验证通过的工具。在DafnyBench程序集上的测试显示,基于LLaMa 3.1 8B的贪心搜索仅成功注解15.7%的方法。由于缺乏大规模训练数据,我们提出一种开放式的合成流程:由大模型生成高阶构思、实现代码,并逐步改进现有程序,再由Dafny进行验证。由此生成的合成数据集DafnySynth被用于增强DafnyBench进行训练。在两个数据集上微调后,LLaMa 8B的成功率提升至50.6%,显著优于基线模型或仅使用单一数据集训练的结果。结果表明,该路径为尚未具备大规模人工标注数据的语言提供了可行的智能助手方案,有望降低用户使用门槛,推动形式化验证的普及。
原文摘要 · Abstract (English)
Formal verification has the potential to drastically reduce software bugs, but its high additional cost has hindered large-scale adoption. While Dafny presents a promise to significantly reduce the effort to write verified programs, users are often required to provide logical annotations to aid the verifier. Here, we explore using a combination of Large Language Models and search to build dafny-annotator: a tool that adds logical annotations to a Dafny method until the verifier can prove it correct. On a test set from the DafnyBench collection of programs, greedy search guided by LLaMa 3.1 8B successfully annotates only 15.7% of the methods. Since this data-driven approach is hindered by the lack of large-scale training data, we propose a method for open-ended synthesis of new Dafny programs in a flexible pipeline where LLMs formulate high-level ideas, implement them, and incrementally propose changes to existing programs, which Dafny validates. This gives us a synthetic dataset, DafnySynth, which we use to augment DafnyBench for training. Fine-tuning on both datasets boosts LLaMa 8B's success rate to 50.6% -- significantly better than the base model, or training on either dataset alone. Our results suggest a path towards capable AI assistants for languages that don't yet have large-scale human-generated examples. In turn, such assistants might reduce friction for users and ultimately drive adoption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。