arXiv:2410.18494cs.SEcs.LG2024-10被引 6

用大模型生成代码时,如何确保代码真正符合用户意图?

Assured Automatic Programming via Large Language Models

  • 通过代码、规范、测试三者协同进化修复,发现真实编程意图
  • 在Dafny数据集上,可验证的自动生成代码比例显著提升
  • 适合关注AI编程可靠性与形式化验证的研究者和开发者

随着基于AI的编程引擎发展,自然语言需求可转化为标准编程语言的可执行代码。然而,AI生成的代码可能不可靠,且原始需求本身可能存在歧义,导致生成代码未能准确反映程序员意图。本文旨在发现程序员的真实意图,并生成符合该意图且带有证明的代码。提出一种名为程序-证明协同进化的新型修复引擎,其修复对象为由大语言模型从同一自然语言描述生成的三元组(代码,逻辑规范,测试)。代码与规范分别捕捉意图的操作性与声明性描述,测试则代表对意图的部分具体理解。通过逐步优化三者间的一致性,实现对用户意图的精确重构。最终获得的形式化逻辑描述被转回自然语言供开发者审查,形成无歧义的意图表达。实验表明,该方法显著提升了在新提出的Dafny语言数据集上可验证的自动代码生成比例。

原文摘要 · Abstract (English)

With the advent of AI-based coding engines, it is possible to convert natural language requirements to executable code in standard programming languages. However, AI-generated code can be unreliable, and the natural language requirements driving this code may be ambiguous. In other words, the intent may not be accurately captured in the code generated from AI-coding engines like Copilot. The goal of our work is to discover the programmer intent, while generating code which conforms to the intent and a proof of this conformance. Our approach to intent discovery is powered by a novel repair engine called program-proof co-evolution, where the object of repair is a tuple (code, logical specification, test) generated by an LLM from the same natural language description. The program and the specification capture the initial operational and declarative description of intent, while the test represents a concrete, albeit partial, understanding of the intent. Our objective is to achieve consistency between the program, the specification, and the test by incrementally refining our understanding of the user intent. Reaching consistency through this repair process provides us with a formal, logical description of the intent, which is then translated back into natural language for the developer's inspection. The resultant intent description is now unambiguous, though expressed in natural language. We demonstrate how the unambiguous intent discovered through our approach increases the percentage of verifiable auto-generated programs on a recently proposed dataset in the Dafny programming language.

大模型编程形式化验证意图发现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。