评估代码生成中问题能否被回答，提升开发效率与准确性。
Assessing the Answerability of Queries in Retrieval-Augmented Code Generation
- 提出评估问题可回答性的新任务，判断查询与API能否生成正确代码。
- 构建了包含1.2万条样本的基准数据集RaCGEval，基线模型准确率仅46.7%。
- 适用于提升LLM在代码生成中的可靠性，适合开发者与模型优化研究者。
得益于大语言模型（LLM）前所未有的语言理解与生成能力，检索增强型代码生成（RaCG）近年来在软件开发中被广泛使用。尽管提升了生产力，但仍频繁出现生成错误代码的情况，尤其当用户查询和提供的API描述无法支持有效答案时，模型仍可能生成看似合理但错误的代码。本研究提出一项评估问题可回答性的新任务，旨在判断在给定查询与检索到的API下，是否能够生成有效代码。为此，我们构建了一个名为检索增强型代码生成可答性评估（RaCGEval）的基准数据集，用于衡量模型在此任务上的表现。实验结果表明，该任务仍处于极高水平挑战，基线模型性能仅为46.7%。此外，本文还探讨了可显著提升性能的方法。
原文摘要 · Abstract (English)
Thanks to unprecedented language understanding and generation capabilities of large language model (LLM), Retrieval-augmented Code Generation (RaCG) has recently been widely utilized among software developers. While this has increased productivity, there are still frequent instances of incorrect codes being provided. In particular, there are cases where plausible yet incorrect codes are generated for queries from users that cannot be answered with the given queries and API descriptions. This study proposes a task for evaluating answerability, which assesses whether valid answers can be generated based on users' queries and retrieved APIs in RaCG. Additionally, we build a benchmark dataset called Retrieval-augmented Code Generability Evaluation (RaCGEval) to evaluate the performance of models performing this task. Experimental results show that this task remains at a very challenging level, with baseline models exhibiting a low performance of 46.7%. Furthermore, this study discusses methods that could significantly improve performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。