用自然语言逐步推导程序状态,提升对代码正确性的验证能力
HoarePrompt: Structural Reasoning About Program Correctness in Natural Language
- 通过类霍尔逻辑的分步推理,让大模型描述程序各阶段状态
- 在竞赛题数据集上,正确率比直接提示提升61%,循环处理增益26%
- 适合需要严谨验证代码符合自然语言需求的研究与工程场景
尽管软件需求常以自然语言表达,但验证程序是否满足这些需求仍是一个困难且未被充分探索的问题。大型语言模型(LLMs)虽有潜力应对该挑战,但实际表现不佳,常无法发现简单错误。为此,我们提出HoarePrompt,一种将程序验证核心思想适配到自然语言的方法。受最强后置条件演算启发,HoarePrompt采用系统化、分步的过程,让LLM生成代码关键点可达程序状态的自然语言描述。针对循环,提出基于少量样本的k-归纳法,借鉴模型检测中的经典技术。在生成状态描述后,利用LLM判断带注释的程序是否符合自然语言需求。为评估分类器在自然语言需求下的正确性判定质量,我们构建了CoCoClaNeL数据集,包含编程竞赛题的解答。实验表明,相比零样本思维链提示,HoarePrompt在多数类别准确率(MCC)上提升61%;相比基于测试生成的分类器,提升106%。归纳推理机制带来26%的MCC增益,凸显其在处理循环时的有效性。
原文摘要 · Abstract (English)
While software requirements are often expressed in natural language, verifying the correctness of a program against such requirements is a hard and underexplored problem. Large language models (LLMs) are promising candidates for addressing this challenge, however our experience shows that they are ineffective in this task, often failing to detect even straightforward bugs. To address this gap, we introduce HoarePrompt, a novel approach that adapts fundamental ideas from program verification to natural language artifacts. Inspired from the strongest postcondition calculus, HoarePrompt employs a systematic, step-by-step process in which an LLM generates natural language descriptions of reachable program states at various code points. To manage loops, we propose few-shot-driven k-induction, an adaptation of the k-induction method widely used in model checking. Once program states are described, HoarePrompt leverages the LLM to assess whether the program, annotated with these state descriptions, conforms to the natural language requirements. For evaluating the quality of classifiers of program correctness with respect to natural language requirements, we constructed CoCoClaNeL, a challenging dataset of solutions to programming competition problems. Our experiments show that HoarePrompt improves the MCC by 61% compared to directly using Zero-shot-CoT prompts for correctness classification. Furthermore, HoarePrompt outperforms a classifier that assesses correctness via LLM-based test generation by an MCC increase of 106%. The inductive reasoning mechanism contributes a 26% boost to MCC, underscoring its effectiveness in managing loops.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。