构建大规模代码依赖推理评测基准，揭示大模型在真实项目中的短板
DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
- 基于581个带测试环境的仓库，覆盖多语言场景
- 顶尖模型仅42.9%的执行通过率，暴露严重缺陷
- 适合关注代码生成可靠性的研究者与工程团队
大型语言模型在自动化软件开发中取得进展，但正确推断依赖关系（即识别项目成功运行所需的内部组件和外部包）仍是挑战。现有研究指出，依赖相关问题导致生成仓库中超过40%的运行时错误。为此，我们提出DI-BENCH，一个大规模基准与评估框架，专门用于评测大模型在依赖推断方面的能力。该基准包含581个仓库，涵盖Python、C#、Rust和JavaScript，并配备测试环境。通过文本与执行层面的综合评估发现，当前表现最佳模型的执行通过率仅为42.9%，表明仍有巨大提升空间。DI-BENCH为评估大模型在完整仓库上的表现提供了新视角，推动更鲁棒的端到端软件合成发展。
原文摘要 · Abstract (English)
Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。