首次在真实GitHub项目中评估开发智能体的补丁效果，发现其表现仍有提升空间。
Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios
- 对比10个顶尖智能体在500个真实问题上的补丁生成能力
- 170个问题未解决，且补丁修改方式与开发者差异明显
- 对简单代码库表现更好，适合用于分解复杂任务的辅助开发
近年来，基于AI的软件工程从预训练模型发展到先进的代理工作流，软件开发智能体代表了下一阶段的重大飞跃。这些能够推理、规划并与外部环境交互的智能体，为复杂软件工程任务提供了潜在解决方案。然而，尽管已有大量研究评估大语言模型生成的代码，针对代理生成补丁（尤其在真实场景下）的系统性研究仍不足。本研究通过评估10个顶级智能体在SWE-Bench Verified数据集上生成的4,892个补丁，分析其对代码质量的影响。结果表明，无单一智能体占据绝对优势，170个问题未能解决，且通过单元测试的补丁在文件和函数修改上与原始开发者补丁存在差异，反映出基准测试用例覆盖不全的问题。多数智能体维持了代码可靠性和安全性，未引入新缺陷或漏洞；部分智能体增加代码复杂度，但多数减少了重复代码和代码异味。此外，智能体在较简单的代码库中表现更优，提示将复杂任务拆解为子任务可提升效果。本研究是首个在真实世界GitHub问题上对智能体补丁的综合性评估，为推进AI驱动的软件开发提供重要参考。
原文摘要 · Abstract (English)
In recent years, AI-based software engineering has progressed from pre-trained models to advanced agentic workflows, with Software Development Agents representing the next major leap. These agents, capable of reasoning, planning, and interacting with external environments, offer promising solutions to complex software engineering tasks. However, while much research has evaluated code generated by large language models (LLMs), comprehensive studies on agent-generated patches, particularly in real-world settings, are lacking. This study addresses that gap by evaluating 4,892 patches from 10 top-ranked agents on 500 real-world GitHub issues from SWE-Bench Verified, focusing on their impact on code quality. Our analysis shows no single agent dominated, with 170 issues unresolved, indicating room for improvement. Even for patches that passed unit tests and resolved issues, agents made different file and function modifications compared to the gold patches from repository developers, revealing limitations in the benchmark's test case coverage. Most agents maintained code reliability and security, avoiding new bugs or vulnerabilities; while some agents increased code complexity, many reduced code duplication and minimized code smells. Finally, agents performed better on simpler codebases, suggesting that breaking complex tasks into smaller sub-tasks could improve effectiveness. This study provides the first comprehensive evaluation of agent-generated patches on real-world GitHub issues, offering insights to advance AI-driven software development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。