评测AI在真实后端开发中全流程编码能力，发现现有模型表现仍不理想。
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
- 构建自动化流水线，从开源项目提取224个真实后端任务。
- 要求模型完成从代码探索到容器化部署的完整流程并通过接口测试。
- 揭示当前大模型在复杂工程任务中仍存在显著能力短板。
大型语言模型演变为自主代理后，其编程能力已从局部代码生成拓展至涉及仓库级、执行驱动的复杂问题求解。然而，现有评估基准多聚焦静态代码逻辑，忽视了真实工程中的动态全流程需求，尤其是后端开发对环境配置和服务部署的严格要求。为此，我们提出ABC-Bench，一个专为评估代理在真实可执行工作流中进行后端编程而设计的基准。通过可扩展的自动化管道，我们从开源仓库中收集了涵盖8种语言和19个框架的224个实际任务。与以往评估不同，ABC-Bench要求代理从仓库探索开始，完成代码编写、容器化部署，并通过外部端到端API测试。大规模评估显示，即使最先进的模型在这些综合性任务上也难以实现可靠表现，凸显当前模型能力与实际后端工程需求之间的巨大差距。代码已公开于https://github.com/OpenMOSS/ABC-Bench。
原文摘要 · Abstract (English)
The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。