用持续集成流程评估代码维护能力，看AI能否长期保持代码质量。
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
- 基于真实仓库的持续集成循环构建动态评测基准
- 100个任务平均跨度233天、71次提交，考验长期迭代能力
- 适合关注AI代码生成可持续性的研究者与工程团队
大型语言模型驱动的智能体在静态漏洞修复等任务中表现优异，但在真实世界中，成熟软件的开发依赖于复杂需求变更和长期功能迭代——这超出了传统一次性修复范式的覆盖范围。为此，我们提出SWE-CI，首个基于持续集成循环的仓库级评测基准，旨在将代码生成评估从静态短期功能正确性，转向动态长期可维护性。核心思想是：通过追踪功能正确性随时间的变化来揭示可维护性。该基准包含100个任务，均来自真实代码仓库，平均开发周期为233天，累计71次连续提交。SWE-CI要求智能体通过数十轮分析与编码迭代系统性完成任务，为评估智能体在长期演化中维持代码质量的能力提供了关键洞察。
原文摘要 · Abstract (English)
Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability. The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。