测试编码智能体在长期代码演化中的综合能力
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- 基于开源项目发布日志构建多文件、多步骤任务集
- 主流模型在复杂任务上准确率仅25%，远低于短任务表现
- 提出新评估指标Fix Rate，衡量部分进展
现有编码智能体评测集中于孤立的单任务，如修复一个错误或添加小功能。但真实软件工程是长期演进过程：开发者需理解高层需求，跨多个文件协调修改，并在多轮迭代中保持功能完整。我们提出SWE-EVO，用于评估这一长期代码演化挑战。该基准源自七个成熟Python开源项目的发布日志，包含48个任务，平均涉及21个文件，每个任务通过平均874个测试用例验证。实验显示显著能力差距：GPT-5.4配合OpenHands在SWE-EVO上仅达25%准确率，而GPT-5.2在SWE-Bench Verified上达72.80%。我们还提出Fix Rate，用于衡量复杂长周期任务中的部分进展。
原文摘要 · Abstract (English)
Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。