让AI同时规划代码和证明,提升正确生成率。
P$^{3}$: Joint Program-and-Proof Planning for Verified Code Generation
- 先制定代码与证明的统一计划,再分步实现。
- 在多个基准上解决率最高,效率提升超37%。
- 适合需要高可靠性的程序自动生成场景。
可验证代码生成要求大语言模型(LLM)同时生成可执行程序和机器可检查的证明,确保程序符合形式化规范,实现正确性构建。现有主流流程将问题拆分为两步:先生成程序,再尝试证明其正确性。我们发现这种顺序方法在实践中常低效且无效。生成程序时若不考虑其可证明性,可能导致细微错误或结构上难以验证,迫使LLM陷入反复修补代码与证明的脆弱循环。受戴克斯特拉观点启发——程序与其正确性论证应协同设计,我们提出P³,一种基于LLM的代理工作流:首先从规范中推导出统一的程序与证明计划,再在此共同计划下展开实现与证明框架。为评估真实场景下的可验证代码生成,我们进一步构建了Lean4Commit0,一个基于真实软件仓库提取核心API并将其需求(包括跨API的逻辑关系)转化为Lean任务的库级基准。使用四个前沿LLM后端,在Verina、AlgoVeri及Lean4Commit0基准上评估P³,其在所有基准-模型组合中均达到最高求解率;相比更强基线,求解率提升4.6–11.2个百分点,每任务API开销降低约40%,壁钟时间减少约37%(在各基准困难子集上)。定向消融实验表明,相比仅规划实现的方案,联合规划带来3.3–8.3个百分点的增益,验证了联合规划的有效性。
原文摘要 · Abstract (English)
Verified code generation asks a large language model (LLM) to generate both an executable program and a machine-checkable proof that the program meets a formal specification, promising software that is correct by construction. The de facto workflow decouples the two halves of the problem: first synthesize a program, then attempt to prove it correct. We observe that this sequential pipeline can be both ineffective and inefficient in practice. A program generated without anticipating its proof can be subtly incorrect or structurally difficult to verify, forcing the LLM into brittle repair loops that alternate between patching the code and patching the proof. Inspired by Dijkstra's view that a program and its correctness argument should be developed hand in hand, we propose $P^3$, an LLM-based agentic workflow that first derives a unified program-and-proof plan from the specification, then elaborates the implementation and proof scaffold under this shared plan. To evaluate verified code generation in realistic settings, we further introduce Lean4Commit0, a repository-derived, library-level benchmark built by extracting core APIs from real-world software repositories and translating their requirements, including relational specifications across APIs, into Lean tasks. Using four frontier LLM backends, we evaluate $P^3$ on Verina, AlgoVeri, and our Lean4Commit0 benchmark, where it achieves the highest solve rate in every benchmark--model setting. Compared with the stronger baseline, it improves solve rates by 4.6--11.2 percentage points and reduces per-task API cost by up to roughly 40\% and wall-clock time by up to roughly 37\% on the difficult subset of each benchmark. A targeted ablation further shows gains of 3.3--8.3 points over implementation-only planning, isolating the benefit of planning the program and proof jointly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。