评测大模型理解代码仓库依赖关系的能力，发现性能差距显著。
DependEval: Benchmarking LLMs for Repository Dependency Understanding
- 构建分层基准，评估模型对依赖、结构和多文件编辑的理解
- 覆盖15,576个真实仓库，8种语言，3项核心任务
- 适合研究代码理解与工程化应用的大模型开发者
尽管大语言模型在代码生成方面展现出巨大潜力，但真实软件开发需要更高级的仓库级推理能力，包括理解依赖关系、项目结构及处理多文件变更。然而，大模型在有效理解和处理复杂代码仓库方面仍缺乏充分探索。为此，我们提出一个分层基准（DependEval），用于评估仓库依赖理解能力。该基准基于从真实网站收集的15,576个仓库，涵盖8种编程语言，评估模型在三个核心任务上的表现：依赖识别、仓库构建和多文件编辑。对超过25个大模型的评估揭示了显著的性能差距，并为仓库级代码理解提供了宝贵洞见。
原文摘要 · Abstract (English)
While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning. This includes understanding dependencies, project structures, and managing multi-file changes. However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored. To address challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding (DependEval). Benchmark is based on 15,576 repositories collected from real-world websites. It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories. Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repository-level code understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。