arXiv:2604.03035cs.SEcs.AI2026-04被引 2

提出新评估框架,让代码智能体在连续任务中测试真实开发能力。

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

  • 构建自动化生成工具,生成包含依赖关系的长序列编码任务
  • 发现孤立任务评估性能虚高最高达20个百分点
  • 揭示模型虽能修复问题但会增加技术债,适合长期开发场景研究者

现有代码智能体数据集仅评估孤立的单个拉取请求(PR)任务,无法反映真实软件开发中代码累积、技术债积累和测试套件增长的实际情况。为此,我们提出一种自动化编码任务生成框架,构建了新数据集SWE-STEPS,通过两种贴近实际开发者工作流的设定评估智能体:对话式迭代请求与单次项目需求文档(PRD)编码。不同于以往将多个独立的拉取请求分开评估的方式,我们的框架考察跨依赖的连续任务执行、回归验证及长期仓库健康度。研究发现,传统孤立任务评估导致成功率虚高,最高达20个百分点,因忽略了前序低效或有缺陷代码的‘溢出’影响。此外分析显示,即使智能体成功解决问题,其生成代码的认知复杂度和技术债仍高于人类开发者,凸显多维度评估的重要性。

原文摘要 · Abstract (English)

Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.

代码生成智能评估开发演化技术债

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。