测试编码智能体在多轮迭代中持续维护代码的能力，发现多数表现不佳。
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
- 设计26个持续性编码任务，每轮保留工作区并验证前后需求
- 13个智能体平均仅50%成功率，第5轮通过率不足首轮一半
- 揭示强弱智能体不同失败模式，适合评估长期协作能力
编码智能体日益作为迭代开发伙伴使用，但现有基准多仅评估单次需求后的最终结果。本文提出EvoCode-Bench，包含26个有状态编码任务和227轮评估。每项任务保留智能体工作区5-15轮，通过可观察行为表达需求，并用累积可执行测试验证新需求与旧功能是否仍有效。评估13个编码智能体，采用两个指标：MT@4（四次尝试后停止的多轮得分）和SR（基于参考完成状态的单轮得分）。多数智能体的SR比MT@4高22-40分。排名也发生变化：最高SR智能体（78.9）在持久执行中仅排第三（MT@4=44.0）。最强智能体多轮成功率约50%，到第5轮整体通过率已低于首轮的一半。故障分析显示：弱智能体早期失败，强智能体则暴露出需求追踪与回归缺陷。数据集与Harbor多轮基础设施已开源。
原文摘要 · Abstract (English)
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。