评测编程智能体在长期迭代任务中代码质量如何恶化
SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
- 设计动态演化任务,让智能体持续扩展自身代码
- 发现所有智能体均无法全程解决任务,代码冗余度升至2.3倍
- 适合研究代码生成鲁棒性或智能体长期决策的学者
软件开发具有迭代性,但现有代理编码评测多采用单次提交设定,掩盖了设计缺陷。近期迭代评测虽有改进,却过度限制智能体的设计空间,无法真实反映其决策对后续扩展的影响。我们提出SlopCodeBench,包含36个问题和196个检查点,要求智能体反复扩展自身解决方案。与以往不同,其动态需求强制架构决策,但内部结构由智能体自主决定。我们测量两种退化现象:结构性侵蚀(复杂度集中)和冗余性(重复代码)。评估15个开源与闭源模型后发现,无一能完整解决任一问题,最佳模型仅通过14.8%检查点。质量随检查点下降:77%轨迹出现结构性侵蚀,75.5%出现冗余。相较473个开源Python仓库,智能体代码冗余度高2.3倍、侵蚀程度高2.0倍,且人类代码退化频率更低、幅度更小。显式质量指导可减少初始冗余与侵蚀达三分之一,但不影响退化速率。SlopCodeBench首次量化了迭代扩展下的代码退化,揭示智能体虽能通过检查点,但生成代码随轮次不断恶化与膨胀。
原文摘要 · Abstract (English)
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。