构建更贴近真实开发的编程智能体评测基准
OmniCode: A Benchmark for Evaluating Software Engineering Agents
- 涵盖四大类任务，覆盖代码修复、测试生成等真实开发场景
- 1794个任务跨三语言，人工验证避免问题模糊，防止数据泄露
- 揭示现有智能体在测试生成和多语言任务中的明显短板
LLM驱动的编程智能体正在重塑现实软件开发方式。为推动研究向更优编程智能体发展，亟需能严格评估其执行各类软件工程任务能力的挑战性基准。然而，现有主流基准如HumanEval和SWE-Bench仅聚焦于竞赛编程或补丁生成等狭窄任务。现实中，软件工程师需处理更广泛的开发任务。为此，我们提出OmniCode，一个新型软件工程基准，包含超越代码或补丁生成的更广、更多样化的任务类别。OmniCode共包含1794个任务，覆盖Python、Java和C++三种编程语言，分为四大核心类别：缺陷修复、测试生成、代码审查修复和风格修复。与以往基准不同，OmniCode任务经人工验证以消除问题定义不清，且通过合成构造或近期整理数据以避免数据泄露，提出一种从有限真实数据中合成多样化软件任务的新框架。我们用SWE-Agent等主流智能体框架评估OmniCode，发现尽管其在Python缺陷修复上表现尚可，但在测试生成及C++、Java等语言任务上显著不足。例如，SWE-Agent在C++测试生成任务上最高仅达25.0%（基于DeepSeek-V3.1）。OmniCode旨在成为稳健的评测基准，推动开发能在多方面均表现优异的编程智能体。代码与数据已开源：https://github.com/seal-research/OmniCode。
原文摘要 · Abstract (English)
LLM-powered coding agents are redefining how real-world software is developed. To drive the research towards better coding agents, we require challenging benchmarks that can rigorously evaluate the ability of such agents to perform various software engineering tasks. However, popular coding benchmarks such as HumanEval and SWE-Bench focus on narrowly scoped tasks such as competition programming and patch generation. In reality, software engineers have to handle a broader set of tasks for real-world software development. To address this gap, we propose OmniCode, a novel software engineering benchmark that contains a broader and more diverse set of task categories beyond code or patch generation. Overall, OmniCode contains 1794 tasks spanning three programming languages - Python, Java, and C++ - and four key categories: bug fixing, test generation, code review fixing, and style fixing. In contrast to prior software engineering benchmarks, the tasks in OmniCode are (1) manually validated to eliminate ill-defined problems, and (2) synthetically crafted or recently curated to avoid data leakage issues, presenting a new framework for synthetically generating diverse software tasks from limited real-world data. We evaluate OmniCode with popular agent frameworks such as SWE-Agent and show that while they may perform well on bug fixing for Python, they fall short on tasks such as Test Generation and in languages such as C++ and Java. For instance, SWE-Agent achieves a maximum of 25.0% with DeepSeek-V3.1 on C++ Test Generation. OmniCode aims to serve as a robust benchmark and spur the development of agents that can perform well across different aspects of software development. Code and data are available at https://github.com/seal-research/OmniCode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。