新基准测试显示大模型仍无法胜任真实代码开发任务
Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
- 构建包含真实项目依赖的980个函数生成任务
- 10个大模型最高仅30%通过率，远低于人类水平
- 检索增强生成优于直接使用函数依赖上下文
近期出现多个仓库级代码生成评测基准（如CoderEval、DevEval、RepoEval、RepoBench和LongCodeArena），用于评估大语言模型（LLMs）在超越独立基准（如HumanEval和MBPP）之外的能力。然而，这些基准大多基于短片段补全、合成样例或局限于小规模仓库，无法反映真实编码任务。为此，我们提出REPOCOD，一个包含真实大型项目复杂任务与合理评估指标的Python代码生成基准。该基准涵盖11个流行项目中的980个完整函数生成任务，其中50.8%需仓库级上下文；每个实例配有314个开发者编写的测试用例以实现更准确评估。我们在REPOCOD上评估了10个大模型，发现其最高仅达到30% pass@1，表明当前模型仍无法有效支持真实软件开发。此外，检索增强生成的表现优于仅使用目标函数依赖作为上下文的方法。
原文摘要 · Abstract (English)
Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。