arXiv:2606.26490cs.SEcs.AI2026-06

用大模型生成分离逻辑验证器的规范,验证成功率仅三成但行为保真度超91%。

An Empirical Study of LLM-Generated Specifications for VeriFast

论文配图:An Empirical Study of LLM-Generated Specifications for VeriFast
图 1 · 摘自论文原文
  • 测试8种提示策略、10个大模型,用正式契约提升生成质量
  • 生成代码和规范功能保真度超91%,但验证成功率为31.4%
  • 94%错误源于对分离逻辑验证器领域知识理解不足,适合验证研究者参考

静态验证工具能保障工业级软件质量,但编写规格说明需大量人工投入。这尤其体现在基于分离逻辑(SL)的验证器上,它们擅长验证堆操作程序,却需大量复杂辅助规格来推理堆结构。近期研究尝试用大语言模型(LLMs)生成代码、测试和证明,包括验证器所需的规格,但多针对非分离逻辑验证器。本文系统评估了LLMs在生成303个C函数规格时的表现,这些函数由分离逻辑验证器VeriFast验证。研究涵盖8种提示方法、10个大模型及三种输入类型,分两阶段进行。通过定量与定性分析,评估生成代码与规格的功能行为、可验证性及错误。结果表明,LLMs在源码与规格的功能保真度均超过91%,但验证成功率仅为31.4%。使用Gemini 2.5 Pro并提供正式契约可提高成功率。此外,94%的错误源于模型对VeriFast等分离逻辑验证器的领域知识理解偏差。研究为优化面向分离逻辑验证器的大模型生成规格提供了实证指导。

原文摘要 · Abstract (English)

Static verification tools can assure industrial scale software, but require significant human labor to write specifications. This is particularly true of static verifiers based on separation logic (SL verifiers), which excel at verifying heapmanipulating programs, but require many complex auxiliary specifications to reason about heap structure. Recent work applies large language models (LLMs) to generate code, tests, and proofs, including specifications for verifiers, but mostly targeting non-SL verifiers. To address this gap, this paper thoroughly evaluates how well LLMs perform when prompted to generate specifications for verifying 303 C functions with the SL verifier VeriFast. We explored eight prompting approaches, ten LLMs, and three input types in two stages. Quantitative and qualitative analyses are used to assess the LLM-generated code and specifications for functional behavior, verifiability and errors. The results show that LLMs preserve functional behavior in source code and specifications (both over 91%), but achieve modest verification success (31.4%). Using Gemini 2.5 Pro and providing formal contracts lead to higher success rates in our setting. Moreover, most errors (94%) come from LLMs' mistakes in the domainspecific knowledge of SL verifiers such as VeriFast. These findings provide guidance for optimizing LLM-generated specifications for SL verifiers.

大模型生成形式化验证分离逻辑VeriFast

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。