用真实开发流程评测大模型修代码能力，模拟提交与测试闭环。
SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving
- 将大模型设为提交者和评审者，模拟真实代码迭代流程。
- 基于400个真实GitHub问题测试，GPT-4o更擅生成补丁，DeepSeek/Gemini更重测试通过率。
- 支持多语言长上下文，适配实际开发场景的模型评估。
我们提出SwingArena，一个贴近真实软件开发流程的大语言模型（LLM）竞争性评估框架。不同于传统静态基准，SwingArena通过将LLM配对为提交者（生成补丁）和评审者（设计测试用例并经持续集成CI管道验证），模拟软件迭代的协作过程。为支持交互式评估，引入检索增强代码生成（RACG）模块，从大型代码库中高效提取语法和语义相关的代码片段，支持C++、Python、Rust和Go等多语言，缓解长上下文挑战，在遵守令牌限制的同时实现跨任务与上下文的可扩展性。实验基于2,300个问题池中筛选出的400个高质量真实GitHub问题进行，结果表明GPT-4o在激进补丁生成上表现优异，而DeepSeek和Gemini更注重CI验证中的正确性。SwingArena为在真实、以CI驱动的开发环境中评估LLMs提供了可扩展且可扩展的方法论。更多详情见项目页：swing-bench.github.io
原文摘要 · Abstract (English)
We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs) that closely mirrors real-world software development workflows. Unlike traditional static benchmarks, SwingArena models the collaborative process of software iteration by pairing LLMs as submitters, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines. To support these interactive evaluations, we introduce a retrieval-augmented code generation (RACG) module that efficiently handles long-context challenges by providing syntactically and semantically relevant code snippets from large codebases, supporting multiple programming languages (C++, Python, Rust, and Go). This enables the framework to scale across diverse tasks and contexts while respecting token limitations. Our experiments, using over 400 high-quality real-world GitHub issues selected from a pool of 2,300 issues, show that models like GPT-4o excel at aggressive patch generation, whereas DeepSeek and Gemini prioritize correctness in CI validation. SwingArena presents a scalable and extensible methodology for evaluating LLMs in realistic, CI-driven software development settings. More details are available on our project page: swing-bench.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。