新基准SWE-Explore评估代码智能体的仓库探索能力，更精准衡量其找代码的能力。
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
- 设计新评测框架，聚焦代码定位与上下文检索能力，避免笼统判断修复成败。
- 覆盖848个问题、10种语言、203个开源仓库，基于成功修复路径生成逐行真实答案。
- 发现顶尖智能体在精准定位和高效排序上显著领先传统方法，尤其在细粒度层级。
现有仓库级编码评测（如SWE-bench）将任务简化为是否修复的二元判断，忽略了代码理解、上下文检索、代码定位和错误诊断等细粒度能力。本文提出SWE-Explore，一个专门评估代码智能体仓库探索能力的新基准。给定仓库和问题，要求智能体在固定行数预算内返回相关代码区域的排序列表。该基准涵盖848个问题，涉及10种编程语言和203个开源仓库。每项任务的逐行真实答案来自独立成功解决该问题的智能体轨迹，提炼出实际访问的代码区域。评测从覆盖率、排序质量与上下文效率三个维度展开，结果显示这些指标与下游修复行为高度相关。在多种检索方法、通用编码代理和专用定位器中，智能体探索者整体表现优于传统检索方案。尽管文件级定位已较成熟，但行级覆盖率与高效排序仍是区分当前最先进探索者的核心维度。
原文摘要 · Abstract (English)
Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。