测试编码代理在科学软件修复中的表现，发现成功率普遍低于50%。
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
- 构建涵盖20个领域的119个科学软件任务基准
- 最佳模型修复成功率不足50%，失败多因科学知识缺失
- 科学知识若不匹配会拖累修复效果，需精准引导
软件正日益成为科学仪器的一部分，科学代码的缺陷不仅影响程序运行，更可能动摇科学结论的证据基础。现有编码代理评估多关注整体任务成功率，难以揭示其在修复科学软件时的失败原因。我们提出SWE-bench Science，一个面向科学软件工程的仓库级基准，包含来自98个GitHub仓库、覆盖20个科学领域的119个任务，按问题驱动、专家探索和工程集成三类范式组织。即使表现最优的Claude Code with Opus-5（max）模型，其pass@1也低于50%，凸显科学软件工程的严峻挑战。分析识别出四类常见失败机制：科学知识或抽象能力不足、探索方向错误或仅做表面修复、修复覆盖不全或系统集成失败、无法泛化已有科学知识。进一步的配对消融实验表明，去除显式科学指导但保留仓库与可执行工程上下文后，科学知识并非始终有益：合理信息可约束修复路径、提升平均性能与令牌效率；而偏差引导则引发锚定效应，未必提升精确修复成功率。SWE-bench Science为研究编码代理在科学软件工程中的能力与失效机制提供了广泛测试平台。
原文摘要 · Abstract (English)
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。