剖析SWE-Bench榜单，揭秘大模型修复系统的真实架构与来源
Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems
- 系统分析79个Lite和99个Verified提交，覆盖80种不同设计
- 发现主流使用私有大模型（如Claude 3.5），且多为智能体架构
- 贡献者涵盖个人开发者到科技巨头，透明度仍存缺口
自动化程序修复（APR）的快速进展得益于人工智能技术，尤其是大语言模型（LLMs）和基于智能体的系统。SWE-Bench 是一项新基准，通过从12个流行开源Python仓库中提取真实问题和拉取请求，评估基于LLM的修复系统。其公开排行榜——SWE-Bench Lite 和 SWE-Bench Verified——已成为追踪进展和比较方案的核心平台。然而，由于提交过程不要求详细文档，许多解决方案的架构设计和来源仍不清晰。本文首次对所有SWE-Bench Lite（79项）和Verified（99项）提交进行全面研究，分析80种独特方法，涵盖提交者类型、产品可用性、LLM使用及系统架构等维度。研究发现，私有LLM（特别是Claude 3.5）占据主导地位，存在智能体与非智能体设计并存现象，贡献者范围从个人开发者到大型科技公司。
原文摘要 · Abstract (English)
The rapid progress in Automated Program Repair (APR) has been driven by advances in AI, particularly large language models (LLMs) and agent-based systems. SWE-Bench is a recent benchmark designed to evaluate LLM-based repair systems using real issues and pull requests mined from 12 popular open-source Python repositories. Its public leaderboards -- SWE-Bench Lite and SWE-Bench Verified -- have become central platforms for tracking progress and comparing solutions. However, because the submission process does not require detailed documentation, the architectural design and origin of many solutions remain unclear. In this paper, we present the first comprehensive study of all submissions to the SWE-Bench Lite (79 entries) and Verified (99 entries) leaderboards, analyzing 80 unique approaches across dimensions such as submitter type, product availability, LLM usage, and system architecture. Our findings reveal the dominance of proprietary LLMs (especially Claude 3.5), the presence of both agentic and non-agentic designs, and a contributor base spanning from individual developers to large tech companies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。