首个多语言代码修复基准，覆盖7种编程语言。
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
- 构建跨7种语言的代码修复评测集，覆盖Java、TypeScript等。
- 包含1632个高质量修复实例，由68名专家标注验证。
- 开源数据生产流程与4723个训练数据，支持强化学习研究。
代码修复任务旨在修改代码库以生成修复给定问题的补丁。然而，现有基准如SWE-bench几乎仅聚焦于Python，难以评估大语言模型在多样化软件生态中的表现。为此，我们提出一个多语言代码修复基准——Multi-SWE-bench，涵盖Java、TypeScript、JavaScript、Go、Rust、C和C++。该基准共包含1632个高质量实例，从2456个候选样本中经68名专家精心标注而成，确保评估的准确性和可靠性。基于此，我们采用三种代表性方法（Agentless、SWE-agent、OpenHands）对一系列先进模型进行评估，并提供全面的实证分析。此外，我们启动了Multi-SWE-RL开源社区，旨在构建大规模强化学习训练数据集。作为初始贡献，我们发布了4723个结构良好的实例，覆盖七种编程语言，为该领域强化学习研究奠定基础。更重要的是，我们开源了完整的数据生产流程及详细教程，鼓励社区持续贡献与扩展数据集。我们期望Multi-SWE-bench与不断发展的Multi-SWE-RL社区能推动强化学习迈向其全部潜力，助力通用人工智能的实现。
原文摘要 · Abstract (English)
The task of issue resolving is to modify a codebase to generate a patch that addresses a given issue. However, existing benchmarks, such as SWE-bench, focus almost exclusively on Python, making them insufficient for evaluating Large Language Models (LLMs) across diverse software ecosystems. To address this, we introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench, covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a total of 1,632 high-quality instances, which were carefully annotated from 2,456 candidates by 68 expert annotators, ensuring that the benchmark can provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we evaluate a series of state-of-the-art models using three representative methods (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with key empirical insights. In addition, we launch a Multi-SWE-RL open-source community, aimed at building large-scale reinforcement learning (RL) training datasets for issue-resolving tasks. As an initial contribution, we release a set of 4,723 well-structured instances spanning seven programming languages, laying a solid foundation for RL research in this domain. More importantly, we open-source our entire data production pipeline, along with detailed tutorials, encouraging the open-source community to continuously contribute and expand the dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL community as catalysts for advancing RL toward its full potential, bringing us one step closer to the dawn of AGI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。