评测编码智能体找文件的能力，发现现有方法仍有明显提升空间。
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
- 基于真实开发流程构建文件检索基准，按任务需求定义相关性。
- 多模型对比显示无单一最优方案，最佳模型在不同指标上表现各异。
- 实际日志轨迹常遗漏关键文件，且选择性拒绝机制效果不佳。
现代编码智能体通常以最终生成正确补丁作为评估标准，但补丁生成依赖于前期的上下文获取阶段——即找到完成任务所需的仓库文件。本文提出 Agent Retrieval Bench，一个面向文件级检索的上游基准。样本源自真实编码工作流信号，评估基于冻结的基线提交仓库，相关性由智能体下一步所需内容决定，而非直接查询与文件的语义相似度。基准涵盖四类正样本检索任务：code2test、comment2context、trace2code、edit2ripple；第五类子集通过自然证据支持的无金标案例和反事实错误仓库控制，评估选择性检索能力。该基准共包含427个样本，覆盖25个仓库：345个正例、50个自然无金标例、32个反事实控制例。数据集包含308个基线提交快照、39.2万文件和790万文本块。我们评估了词法检索、RepoMap、开源嵌入模型、选择性拒答及日志轨迹中的上下文选取。无单一检索方法占优：Qwen3-Embedding-4B在正样本上样本加权MRR最高，Qwen3-Embedding-8B Recall@20表现最佳，RepoMap在8K token预算下上下文产出率最高，各任务胜者差异显著。以反事实控制校准的选择性阈值，在自然无金标案例上未能提升成功率，暴露校准差距。日志轨迹在27%–35%样本中完全遗漏金标文件。受控种子干预实验表明，基于检索获得的初始上下文能带来更高的文件F1值，且后续探索更少；而理想金标上下文仍存在显著提升空间。
原文摘要 · Abstract (English)
Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。