评测AI在代码版本管理任务上的表现，填补了现有基准的空白。
GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git
- 构建涵盖三种语言的Git操作任务基准，覆盖实际开发场景。
- 使用GPT-4o工具链在120样本上达到21.11%解决率。
- 适合研究智能编程助手与全栈开发自动化方向的学者。
现有软件工程AI代理评估基准（如SWE-bench）虽推动了编程能力发展，但忽略了版本控制系统（VCS）等关键开发流程。为此，我们提出GitGoodBench，一个针对Git操作任务的新基准。该基准基于宽松开源的Python、Java和Kotlin仓库提取出三个核心场景，包含三个数据集：完整评估集（900个样本）、快速原型集（120个样本）和训练语料库（17,469个样本）。我们以配备自定义工具的GPT-4o为基线，在原型集上实现21.11%的整体解决率。我们期望GitGoodBench能成为推动具备完整开发能力的SE智能体发展的关键一步。
原文摘要 · Abstract (English)
Benchmarks for Software Engineering (SE) AI agents, most notably SWE-bench, have catalyzed progress in programming capabilities of AI agents. However, they overlook critical developer workflows such as Version Control System (VCS) operations. To address this issue, we present GitGoodBench, a novel benchmark for evaluating AI agent performance on VCS tasks. GitGoodBench covers three core Git scenarios extracted from permissive open-source Python, Java, and Kotlin repositories. Our benchmark provides three datasets: a comprehensive evaluation suite (900 samples), a rapid prototyping version (120 samples), and a training corpus (17,469 samples). We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall. We expect GitGoodBench to serve as a crucial stepping stone toward truly comprehensive SE agents that go beyond mere programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。