评测编程大模型从需求到完整项目开发的全流程能力
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
- 构建端到端项目开发评测基准，覆盖设计与迭代
- 六款模型综合通过率仅27.38%，复杂系统设计能力弱
- 适合评估智能编程工具在真实项目中的实用性
最近的编程代理可从简单提示生成完整代码库，但现有评估多聚焦于单个问题修复，难以反映全流程开发能力。我们提出ProjDevBench，一个端到端的评测基准，向编程代理提供项目需求并评估生成的代码仓库。结合在线判题（OJ）测试与大模型辅助代码审查，该基准从系统架构设计、功能正确性及迭代优化三方面评估代理表现。我们整理了涵盖8类任务的20个编程问题，包括概念型与真实应用场景。对6个基于不同LLM后端的编程代理进行评估，结果显示总体通过率为27.38%：模型能处理基础功能与数据结构，但在复杂系统设计、时间复杂度优化与资源管理方面表现不佳。评测基准已开源：https://github.com/zsworld6/projdevbench。
原文摘要 · Abstract (English)
Recent coding agents can generate complete codebases from simple prompts, yet existing evaluations focus on issue-level bug fixing and lag behind end-to-end development. We introduce ProjDevBench, an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories. Combining Online Judge (OJ) testing with LLM-assisted code review, the benchmark evaluates agents on (1) system architecture design, (2) functional correctness, and (3) iterative solution refinement. We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios, and evaluate six coding agents built on different LLM backends. Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality and data structures but struggle with complex system design, time complexity optimization, and resource management. Our benchmark is available at https://github.com/zsworld6/projdevbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。