通过双向注释交互提升代码生成准确率,让开发者与大模型更好对齐意图。
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
- 用可编辑注释作为桥梁,实现代码与注释的双向迭代优化
- 在HumanEval上使code-davinci-002的pass@1提升17.1%
- 用户实测效率提高16.7%,任务成功率提升10.5%,适合工具开发与协作场景
大型语言模型(LLMs)在代码生成方面展现出前所未有的能力,但生成的代码仍存在大量功能错误,尤其在模型未见过的复杂任务中。近期研究显示,开发者难以检查和修复大模型生成的错误代码,降低了生产力与信任度。受沟通中的相互对齐理论启发,我们提出一种交互式方法,以代码注释为媒介,促进开发者与大模型建立共同理解。该方法通过交替进行代码生成、内联注释生成和上下文化用户反馈,实现迭代对齐。我们在两个主流基准上评估了该方法,结果表明其显著提升了多个前沿大模型的表现,例如在HumanEval上使code-davinci-002的pass@1提升17.1%。此外,我们对12名参与者进行了用户研究,对比了两种基线:(1) 使用GitHub Copilot,(2) 使用多轮程序合成范式。使用本方法时,参与者完成任务的速度快16.7%,任务成功率提高10.5%。结果表明,通过交互式注释精炼,能有效建立相互对齐,提升代码生成精度与开发者信心。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated unprecedented capability in code generation. However, LLM-generated code is still plagued with a wide range of functional errors, especially for complex programming tasks that LLMs have not seen before. Recent studies have shown that developers often struggle with inspecting and fixing incorrect code generated by LLMs, diminishing their productivity and trust in LLM-based code generation. Inspired by the mutual grounding theory in communication, we propose an interactive approach that leverages code comments as a medium for developers and LLMs to establish a shared understanding. Our approach facilitates iterative grounding by interleaving code generation, inline comment generation, and contextualized user feedback through editable comments to align generated code with developer intent. We evaluated our approach on two popular benchmarks and demonstrated that our approach significantly improved multiple state-of-the-art LLMs, e.g., 17.1% pass@1 improvement for code-davinci-002 on HumanEval. Furthermore, we conducted a user study with 12 participants in comparison to two baselines: (1) interacting with GitHub Copilot, and (2) interacting with a multi-step code generation paradigm called Multi-Turn Program Synthesis. Participants completed the given programming tasks 16.7% faster and with 10.5% improvement in task success rate when using our approach. Both results show that interactively refining code comments enables the collaborative establishment of mutual grounding, leading to more accurate code generation and higher developer confidence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。