评测代码智能体利用代码库解决真实开发任务的能力,发现当前系统仅能完成近半数任务。
GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
- 构建54个跨领域真实任务,结合代码库与自动化评估流程
- 最佳模型仅解决48.15%任务,最新记录为62.96%
- 引入经济价值指标,衡量成功率、成本与开发者薪资的综合效益
除了从零编写代码外,利用大规模代码库(如GitHub)完成实际任务在真实软件开发中至关重要,但现有基准很少评估代码智能体在此类真实、工作流驱动场景下的表现。为此,我们提出GitTaskBench,一个涵盖7个模态和7个领域的54个真实任务的基准,每个任务均配对相关代码仓库与人工校准的自动化评估框架,明确实用成功标准。除执行与任务成功率外,我们还提出alpha值度量,量化智能体性能的经济收益,综合任务成功率、令牌成本与平均开发者薪资。在三个先进智能体框架及多个顶级LLM上的实验表明,利用代码库解决复杂任务仍具挑战:即使表现最优的OpenHands+Claude 3.7也仅完成48.15%的任务(近期进展中RepoMaster+Claude 3.5创下62.96%的新纪录)。错误分析显示,超过一半失败源于环境配置、依赖解析等看似简单却关键的步骤,凸显亟需更强的工作流管理与超时应对能力。我们开源了GitTaskBench,旨在推动代码感知推理、执行与部署的研究,助力智能体实现端到端真实任务求解。基准与代码已发布于https://github.com/QuantaAlpha/GitTaskBench。
原文摘要 · Abstract (English)
Beyond scratch coding, exploiting large-scale code repositories (e.g., GitHub) for practical tasks is vital in real-world software development, yet current benchmarks rarely evaluate code agents in such authentic, workflow-driven scenarios. To bridge this gap, we introduce GitTaskBench, a benchmark designed to systematically assess this capability via 54 realistic tasks across 7 modalities and 7 domains. Each task pairs a relevant repository with an automated, human-curated evaluation harness specifying practical success criteria. Beyond measuring execution and task success, we also propose the alpha-value metric to quantify the economic benefit of agent performance, which integrates task success rates, token cost, and average developer salaries. Experiments across three state-of-the-art agent frameworks with multiple advanced LLMs show that leveraging code repositories for complex task solving remains challenging: even the best-performing system, OpenHands+Claude 3.7, solves only 48.15% of tasks (recent progress has pushed the frontier further, with RepoMaster+Claude 3.5 achieving a new record of 62.96%). Error analysis attributes over half of failures to seemingly mundane yet critical steps like environment setup and dependency resolution, highlighting the need for more robust workflow management and increased timeout preparedness. By releasing GitTaskBench, we aim to drive progress and attention toward repository-aware code reasoning, execution, and deployment -- moving agents closer to solving complex, end-to-end real-world tasks. The benchmark and code are open-sourced at https://github.com/QuantaAlpha/GitTaskBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。