评测大模型在代码仓库中增量实现新功能的能力
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
- 基于83个GitHub仓库的合并请求构建任务
- 模型需同时完成新代码生成与已有代码修改
- 揭示当前大模型在仓库级开发中的显著短板
在代码仓库层面实现新功能是代码生成模型的重要应用场景。然而，现有基准测试缺乏针对该能力的专门评估框架。为填补这一空白，我们提出FEA-Bench，一个用于评估大语言模型（LLMs）在代码仓库内进行增量开发能力的基准。从83个GitHub仓库收集合并请求，并采用基于规则和意图的过滤方法，构建聚焦于新功能开发的任务实例。每个任务实例包含代码变更，并配对相关单元测试文件，以确保解决方案可验证。特征实现要求模型同时具备新组件的代码补全能力和对代码仓库中其他相关部分的编辑能力，从而提供更全面的大型语言模型自动化软件工程能力评估方式。实验结果表明，大模型在FEA-Bench上的表现显著较差，凸显了在仓库级增量代码开发中面临的重大挑战。
原文摘要 · Abstract (English)
Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs' automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。