arXiv:2608.28281cs.AI2026-08

评测模型如何引导代码代理完成长期任务,提升自动化编程可靠性。

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

论文配图:LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
图 1 · 摘自论文原文
  • 用控制器模型动态指导代码代理的每一步操作,实现任务闭环
  • 全任务成功率最高仅24.69%,显示长程控制仍有巨大提升空间
  • 可显著降低推理成本64.4%,适合评估自动化开发框架的智能性

Loop Engineering 正成为围绕代码代理组织开发工作的新实践。开发者不再手动编写每个提示,而是设计循环机制来监控进度、分配任务、执行检查并决定下一步行动。即使拥有强大的代码代理,循环仍可能因依赖过时信息、跳过验证、预算错配或过早终止而导致失败。然而,单次端到端运行的结果无法区分成功或失败是源于循环策略还是代理能力。为此,我们提出 LoopArena,一个用于评估模型作为控制器引导独立代码代理完成长时间任务的能力的基准。被评估的模型为控制器:每次编码后接收结构化进展摘要,并指令固定不变的代码代理(工作代理)下一步应做什么或验证什么,或决定是否停止。该基准在三种互补场景下评估此能力,分别对应不同的执行范围与成本。类型 I 通过执行验证的问题进行下一步循环合约选择,无需实际运行工作代理;类型 II 在完整任务的选定片段上重复控制;类型 III 从初始状态评估配对的完整任务。在完整任务上,观察到的最佳严格成功率仅为24.69%,表明长周期循环控制仍有巨大改进空间。各控制器间平均估计推理成本降低64.4%,且类型 II 在核心指标上的排序与主标准高度一致(斯皮尔曼ρ=0.9747)。我们已在 https://github.com/AMAP-ML/LoopArena 公开基准数据与评估代码。

原文摘要 · Abstract (English)

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=\textbf{0.9747}\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .

代码生成自动化开发智能控制基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。