评测编程大模型如何找代码上下文，揭示其真实能力短板。
ContextBench: A Benchmark for Context Retrieval in Coding Agents
- 构建1136个任务的上下文检索基准，带人工标注黄金上下文。
- 大模型偏好召回率但准确率低，实际使用上下文不足已探索的40%。
- 适合研究编程代理推理过程、优化上下文利用效率的开发者。
基于大语言模型的编程代理在自动化问题修复基准上表现强劲，但现有评估多关注最终任务成功率，难以揭示代理在求解过程中如何检索与使用代码上下文。本文提出ContextBench，一个面向上下文检索过程的评估基准。该基准包含来自8种编程语言66个仓库的1,136个问题修复任务，每项任务均配有经人工标注的黄金上下文。我们进一步构建了自动化评估框架，可追踪代理行为轨迹并度量上下文召回率、精确率与效率。基于此，我们评估了四个前沿大模型与五个编程代理。结果表明，复杂代理架构仅带来微弱的上下文检索提升（‘编码代理的苦果’），大模型始终更倾向召回而非精确，且探索与实际使用之间存在显著差距。ContextBench通过引入中间黄金上下文指标，补全了端到端评估的盲区，为软件任务中的大模型推理提供关键中间信号。
原文摘要 · Abstract (English)
LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。