arXiv:2410.01353cs.SEcs.AI2024-10被引 22

构建真实开发场景的代码补全评估基准,更贴近开发者实际需求。

Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?

  • 用智能代理自动爬取仓库、生成测试用例,避免数据泄露。
  • 基于工业级数据重构评估标准,聚焦开发者意图与上下文理解。
  • 适合研究代码生成与开发工具的工程师和研究人员使用。

代码补全是提升开发者生产力的关键任务,但现有评估基准多聚焦粗粒度任务,缺乏对真实开发场景的模拟,且依赖耗时的人工标注。本文分析工业级代码补全工具数据,重新定义评估标准,以更好反映开发者意图。基于此,提出 Codev-Agent 智能代理系统,实现仓库自动爬取、执行环境构建、动态调用链提取及新测试用例生成,避免数据泄露。由此构建的 Codev-Bench 是一个细粒度、真实世界、仓库级、开发者中心的评估框架,能检验代码补全工具在多样化上下文中捕捉即时意图并提供恰当建议的能力,为现代软件开发中的代码补全提供更真实的评估基准。

原文摘要 · Abstract (English)

Code completion, a key downstream task in code generation, is one of the most frequent and impactful methods for enhancing developer productivity in software development. As intelligent completion tools evolve, we need a robust evaluation benchmark that enables meaningful comparisons between products and guides future advancements. However, existing benchmarks focus more on coarse-grained tasks without industrial analysis resembling general code generation rather than the real-world scenarios developers encounter. Moreover, these benchmarks often rely on costly and time-consuming human annotation, and the standalone test cases fail to leverage minimal tests for maximum repository-level understanding and code coverage. To address these limitations, we first analyze business data from an industrial code completion tool and redefine the evaluation criteria to better align with the developer's intent and desired completion behavior throughout the coding process. Based on these insights, we introduce Codev-Agent, an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage, ensuring fair and effective comparisons. Using Codev-Agent, we present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Bench assesses whether a code completion tool can capture a developer's immediate intent and suggest appropriate code across diverse contexts, providing a more realistic benchmark for code completion in modern software development.

代码补全评估基准智能代理开发工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。