arXiv:2607.11042cs.SEcs.AI2026-07

用真实后端接口测试大模型端到端写代码能力,发现模型常能实现局部功能却难完成完整服务。

BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

论文配图:BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services
图 1 · 摘自论文原文
  • 以OpenAPI契约和HTTP测试为标准,让模型从零生成可部署的后端服务。
  • 顶级模型仅28.6%任务在最终测试下通过,暴露整体系统构建能力不足。
  • 自洽演化测试与代码修复机制,避免隐藏要求,提升评估可靠性。

大型语言模型(LLMs)正越来越多地用于代理式编程场景,能够检查文件、执行命令、运行测试、观察失败并迭代修正代码。这一转变提出了核心评估问题:一个代理式LLM能否生成既可部署又行为正确的端到端软件?后端服务提供了一个受控但真实的评估基础。其API暴露应用级可执行语义,且通过黑盒HTTP交互可确定性地验证部署行为是否符合OpenAPI契约。我们提出BackendForge,一个由56个真实开源项目重构的合同定义后端生成任务基准。给定可见规范和OpenAPI契约,模型需生成一个可通过HTTP测试构建、部署并评估的Docker化服务。为强化评估而不引入隐藏需求,BackendForge采用测试代理与代码代理协同进化测试断言与参考实现,其中测试代理提出基于规范的后端测试,代码代理修复参考实现。尽管最佳模型GPT-5.5在基础断言下成功率达55.4%,但在最终断言下仅28.6%成功。该差距表明当前LLMs虽能实现多数局部API行为,但仍难以生成完整的后端服务。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution? Backend services provide a controlled but realistic substrate for this evaluation. Their APIs expose application-level executable semantics, and deployed behavior can be checked deterministically against an OpenAPI contract through black-box HTTP interactions. We introduce BackendForge, a benchmark of 56 contract-defined backend generation tasks rewritten from real open-source applications. Given a visible specification and an OpenAPI contract, an LLM must generate a Dockerized service that is built, deployed, and evaluated only through HTTP tests. To strengthen evaluation without introducing hidden requirements, BackendForge uses a test agent and a code agent to co-evolve the test oracle and reference service, where the test agent proposes specification-grounded backend tests and the code agent repairs the reference implementation. Although the best-performing model, GPT-5.5, succeeds on 55.4\% of tasks under the base oracle, it succeeds on only 28.6\% under the final oracle. This gap suggests that current LLMs can implement many local API behaviors, but still struggle to produce complete backend services.

代码生成智能代理评估基准后端服务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。