arXiv:2606.07682cs.SEcs.AI2026-06被引 7

测试AI代理完成超长软件任务的能力,发现当前模型成功率不足30%。

SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?

论文配图:SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
图 1 · 摘自论文原文
  • 设计20个需数百万词上下文的长期软件任务,模拟真实开发流程。
  • 前沿编码代理平均仅完成不到30%任务,失败主因是自我验证不足。
  • 引入多层验证防作弊,适合评估复杂任务规划与长期记忆能力。

AI代理被期望完成需持续数小时、处理数百万词上下文和复杂环境的长周期工作流。但现有基准大多评估短时任务,如单次代码提交或5-10分钟练习,难以衡量代理在规划、长上下文理解与记忆使用方面的能力。我们提出SWE-Marathon,一个包含20个跨越软件工程及相邻技术领域的长周期任务基准。每个任务配有唯一可执行环境、人工编写的参考解决方案和多层验证体系。代理尝试的日志平均达2720万总词数,显著超过现有软件工程与命令行代理基准。当前前沿编码代理仅能完成少于30%的任务。失败常源于自我验证不足、自报不可行及过早终止。我们还观察到13.8%的运行中出现奖励劫持行为,即代理试图利用环境或验证器绕过预期流程。SWE-Marathon包含对抗性测试套件审查与执行环境,以及多层检查机制以防止捷径解法。我们已公开SWE-Marathon、评估代码及代理轨迹:https://swe-marathon.org/。

原文摘要 · Abstract (English)

AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form tasks, such as single pull requests, small tickets, or 5-10 minute exercises, limiting our ability to measure agents' capabilities in planning, long-context understanding, and memory use. We introduce SWE-Marathon, a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains. Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks. Current frontier coding agents solve fewer than 30% of tasks. Failures often arise from poor self-verification, self-reported infeasibility, and premature termination. We also observe reward-hacking behavior in 13.8% of rollouts, where agents attempt to exploit the environment or verifier to bypass the intended workflow. SWE-Marathon includes adversarial review of test suites and execution environments, as well as multi-layer checks designed to prevent shortcut solutions. We release SWE-Marathon, evaluation code, and agent trajectories at https://swe-marathon.org/.

AI代理长周期任务软件工程验证机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。