SWE Atlas评测代码代理在真实开发流程中的工程能力
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- 构建三类真实开发任务的综合评估基准
- 顶尖模型仍难通过复杂边界案例和工程规范检验
- 适合评估代码生成的工程质量而非仅功能正确性
我们提出SWE Atlas，一个涵盖三种专业软件工程流程的基准测试套件：代码库问答（124项任务）、测试编写（90项任务）和重构（70项任务）。与以往基准不同，SWE Atlas聚焦于被忽视但实际重要的任务类别，采用针对性的评估协议，并使用未明确指定的代理式任务形式，更贴近真实使用场景。其评估框架结合程序化检查与评分标准，不仅验证功能正确性，还评估软件工程质量，包括测试与重构的完整性、可维护性、可复用抽象及代码库整洁度。我们在SWE Atlas上评估了多种前沿及开源模型，发现GPT-5.4和Opus 4.7表现最优，而即使最佳开源模型也得分较低。分析表明，顶级模型依赖大规模代码库探索与运行时推理，但仍持续在细微边界情况、复杂运行时分析及工程最佳实践遵守方面表现不佳。SWE Atlas为衡量代码代理的正确性与工程质量提供了互补性评估工具。
原文摘要 · Abstract (English)
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。