评测大模型代码代理在连续包升级中的表现，发现其仍易破坏原有功能。
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
- 构建分治合成流水线，将发布说明与代码差异对齐生成真实可执行升级需求。
- 包含155次版本迭代、1660个需求，主流模型平均准确率仅50.2%。
- 适合关注代码自动化维护与智能代理真实能力的研究者参考。
由大语言模型驱动的代码代理正被期望承担超出孤立问题修复的现实软件维护任务。现有基准虽转向真实软件演化，却极少涵盖以包发布为粒度的持续维护，其中变更成批打包、发布并被后续版本继承。本文提出SWE-Chain，用于评估代理在链式发布级包升级任务上的表现，每个升级依赖前序代码库状态。为生成升级规范，设计了分治合成流水线，将发布说明与代码差异对齐，确保需求基于真实代码变更，对代理有信息量且可实现。SWE-Chain涵盖9个真实Python包的12条升级链，共155个版本过渡和1,660个经验证的升级需求。在九种前沿代理模型配置下，代理在构建+修复环境下平均解决率为44.8%，精确率为65.4%，F1为50.2%；其中Claude-Opus-4.7（Claude Code）表现最佳，分别为60.8%、80.6%和68.5%。结果表明SWE-Chain兼具可行性与区分度，并揭示当前代理在不破坏原有功能的前提下完成链式包升级仍具挑战。
原文摘要 · Abstract (English)
Coding agents powered by large language models are increasingly expected to perform realistic software maintenance tasks beyond isolated issue resolution. Existing benchmarks have shifted toward realistic software evolution, but they rarely capture continuous maintenance at the granularity of package releases, where changes are bundled, shipped, and inherited by subsequent versions. We present SWE-Chain, a benchmark for evaluating agents on chained release-level package upgrades, where each transition builds on the agent's prior codebase. To produce upgrade specifications, we design a divide-and-conquer synthesis pipeline that aligns release notes with code diffs for each version transition, ensuring the requirements are grounded in actual code changes, informative to agents, and feasible to implement. SWE-Chain contains 12 upgrade chains across 9 real Python packages, with 155 version transitions and 1,660 grounded upgrade requirements. Across nine frontier agent-model configurations, agents achieve an average of 44.8% resolving, 65.4% precision, and 50.2% F1 under the Build+Fix regime, with Claude-Opus-4.7 (Claude Code) leading at 60.8% resolving, 80.6% precision, and 68.5% F1. These results show that SWE-Chain is both feasible and discriminative, and reveal that current agents still struggle to make correct upgrades across chained package releases without breaking existing functionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。