新基准SWE-MERA动态评估代码大模型，解决旧数据泄露问题。
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
- 自动抓取真实GitHub问题，持续更新避免数据污染
- 当前有728个高质量样本，约1万个潜在任务
- 适合评估代码生成模型在真实场景下的表现
大语言模型在软件工程领域的快速发展暴露出现有评测基准的严重缺陷，尤其是广泛使用的SWE-bench数据集存在严重数据污染问题：32.67%的成功修复存在直接解法泄露，31.08%因测试用例不充分而通过。为此，我们提出SWE-MERA，一个动态、持续更新的评测基准，通过自动化收集真实GitHub问题并进行严格质量验证，构建可靠评估流程，有效降低污染风险。目前已有728个样本可用，约10,000个潜在任务。使用Aider编码代理评估表明，该基准对先进模型具备强区分能力。我们报告了十余种近期主流大模型在2024年9月至2025年6月间采集任务上的表现。
原文摘要 · Abstract (English)
The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset. Recent studies have uncovered severe data contamination issues, e.g., SWE-bench reports 32.67% of successful patches involve direct solution leakage and 31.08% pass due to inadequate test cases. We introduce SWE-MERA, a dynamic, continuously updated benchmark designed to address these fundamental challenges through an automated collection of real-world GitHub issues and rigorous quality validation. Our approach implements a reliable pipeline that ensures quality while minimizing contamination risks, resulting in approximately 10,000 potential tasks with 728 samples currently available. Evaluation using the Aider coding agent demonstrates strong discriminative power in state-of-the-art models. We report performance across a dozen recent LLMs evaluated on tasks collected between September 2024 and June 2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。