评测大模型在真实代码库中利用上下文生成代码的能力,发现效果仍不理想。
ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation
- 设计新基准ReCUBE,要求模型根据其他文件重建被遮蔽的代码。
- 最先进模型在完整上下文下严格通过率仅37.57%,提升空间大。
- 提出依赖图引导工具CCE,可显著提升模型在代码生成中的表现。
大型语言模型(LLMs)已成为强大的编码助手,可通过代理探索或全上下文生成在大型代码库中工作。现有基准涵盖多种编程能力,如修复GitHub问题,但均未直接衡量模型在代码生成中利用代码库级上下文的有效性。为此,我们提出ReCUBE基准,要求模型在仅使用其余源文件、依赖项说明和文档作为上下文的前提下,重建真实代码库中的被遮蔽文件。ReCUBE采用基于使用场景的测试用例,模拟内部模块逻辑与跨文件集成,反映真实软件使用模式。我们进一步提出基于依赖图的呼叫者中心探索(Caller-Centric Exploration, CCE)工具包,可集成至代理框架中,引导代理在代码库探索中聚焦最相关的调用文件。在四个设置下对八种模型的实验表明,即使是最先进的模型,代码库级上下文利用仍极具挑战,GPT-5在全上下文设置下严格通过率仅为37.57%。引入CCE工具包的代理在所有模型上均优于基线,严格通过率最高提升7.56%。我们已将基准、代码和评估框架开源,供自然语言处理研究社区使用。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。