DeepSWE用全新长任务评估编程智能体，更真实地检验解决问题能力。
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- 从91个开源库原创编写113个长周期任务，避免训练数据泄露。
- 手写验证器评估任意正确实现，错误率仅1.4%，远低于SWE-Bench Pro的32.4%。
- 适合评估前沿编程代理，尤其关注真实工程能力而非记忆复现。
DeepSWE是一个包含113个原始、长周期软件工程任务的基准，用于评估编程智能体。现有公开的智能体编程基准多基于GitHub合并修复挖掘，存在两个问题：修复内容及其讨论可能已在预训练中出现，高分可能反映记忆而非解决能力；每个任务由随合并修复发布的测试评判，这些测试仅针对特定修复方案，无法有效评估其他正确实现或可能误判不完整代码。DeepSWE规避了这些问题：任务由91个活跃开源仓库和五种语言中的原始编写而成，且不回传上游，参考解保持在公共数据之外；每项任务由人工编写的验证器评估，只要实现功能即通过。独立大模型评委对评分结果的分歧率仅为1.4%，远低于SWE-Bench Pro的32.4%。尽管提示长度仅为SWE-Bench Pro的一半，其参考解涉及代码量达5.5倍，且能更清晰地区分前沿智能体的性能差距。我们已发布该基准、验证器及完整评估轨迹。
原文摘要 · Abstract (English)
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。