新基准评估代码代理长期开发能力，聚焦循环工程与依赖管理。
LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
- 构建依赖图任务链，模拟真实长期开发流程
- 最强模型仅解决25%任务，暴露规划与回归缺陷
- 适合评估长期编码代理的稳定性与可维护性
随着代码代理从短周期任务向长期软件开发演进，评估框架也需从测试夹具工程转向循环工程。现有基准多关注局部任务或最终结果，难以反映持续执行过程。我们提出LOOPSBENCH，一个面向长期开发循环工程的评估基准。每个任务由可独立测试的开发单元构成依赖有向无环图（DAG），并带有源码支持的前置依赖边。基准涵盖112个真实来源的任务，覆盖8种编程语言和9个领域。其流程感知运行时按就绪前沿释放测试，并将已完成节点保留为回归义务。我们评估了主流循环实现与前沿代码代理的组合。最优配置（Opus-4.7 + Claude Code + 外层续写）仅完成25.00%的任务。记录的计划仅部分恢复源码重建的前置依赖图，且回归事件在所有评估循环轨迹中仍可观察。我们已开源全部数据与代码，包括所有任务、超5300个开发单元及可执行测试，地址：microsoft/Loopsbench。
原文摘要 · Abstract (English)
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。