测试代码智能体在跨库修复等复杂任务中的表现,发现其仍严重依赖外部知识。
BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
- 构建500个实例的跨仓库基准,涵盖多种真实开发场景
- 最强模型仅达56.65分,多数任务仍难完成
- 检索增强虽有帮助但效果有限,关键在精准整合信息
现有代码智能体评估多集中于单仓库局部问题修复,忽视需外部知识或跨仓库变更的工程任务。我们提出BeyondSWE,一个源自246个真实GitHub仓库的500实例基准,用于评估代码智能体在跨仓库问题修复、领域特定问题、依赖驱动迁移及文档到仓库生成等四类场景下的能力,覆盖更广的知识与操作范围。评估显示,该基准尚未饱和:基于OpenHands的最佳代理平均得分为46.12,而使用GPT-5.4(xhigh)的Codex最强模型在搜索感知提示下达到56.65。为探究外部信息访问是否缩小差距,我们引入SearchSWE作为可控诊断基线。结果显示,搜索访问提升多数模型性能,对部分任务帮助显著,但增益有限且不均衡,表明当前智能体仍难以将检索信息转化为精确、版本兼容且可本地执行的代码修改。这说明深度代码检索仍是开放难题,未来进展需依赖能可靠结合外部证据与仓库内推理和执行验证的智能体。
原文摘要 · Abstract (English)
Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。