构建统一评估框架，全面测试大模型编程能力
SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
- 整合8类任务、8种场景、10门语言的统一评测体系
- 基于2000个真实GitHub提交，覆盖多语言开发流程
- 适合研究智能编程助手和代码生成模型的开发者
评估大语言模型在软件工程中的表现受限于任务覆盖窄、语言偏见和与真实开发流程脱节。现有基准多聚焦算法题或以Python为主的修错任务，忽视了软件工程的关键维度。为此，我们提出SWE-Compass，一个将异构代码任务统一到结构化、贴近生产的评估框架。该基准涵盖8类任务、8种编程场景、10种编程语言，包含2000个从真实GitHub Pull Request中筛选并验证的高质量实例。我们在SWE-Agent和Claude Code两种代理框架下对十款先进LLM进行评测，揭示了任务类型、语言和场景间的明显难度层级。通过贴近真实开发者实践，SWE-Compass为诊断和推进大模型的智能编程能力提供了严谨可复现的基准。
原文摘要 · Abstract (English)
Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, SWE-Compass provides a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。