新基准测试发现代码AI在含图像的前端任务上表现大降
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
- 构建含图像的多模态代码修复数据集，覆盖17个JavaScript库
- 顶尖代码AI在新任务上仅解决6%问题，视觉理解能力严重不足
- 语言无关设计的SWE-agent表现最优，解决12%任务
自主软件工程系统已能修复漏洞和开发功能，但现有评估标准SWE-bench仅基于Python代码库，问题描述以文本为主，缺乏图像等视觉元素。为探究系统在未覆盖领域的泛化能力（如前端、游戏开发、DevOps），我们提出SWE-bench Multimodal（SWE-bench M），用于评估系统在包含图像的JavaScript可视化软件中的缺陷修复能力。该数据集包含来自17个JavaScript库的617个任务实例，涵盖网页界面设计、流程图绘制、数据可视化、语法高亮及交互式地图等功能。每个任务至少包含一张图像，出现在问题描述或单元测试中。分析显示，顶级SWE-bench系统在新任务上表现不佳，暴露出视觉理解与跨语言泛化能力的局限性。最后，SWE-agent凭借其语言无关的灵活特性，在该数据集上解决12%的任务，显著优于次优系统（6%）。
原文摘要 · Abstract (English)
Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。