测试大模型从零构建完整软件的能力，发现当前模型难以真正独立开发项目。
ProgramBench: Can Language Models Rebuild Programs From Scratch?
- 基于程序文档和行为测试，评估模型从零构建完整代码库的能力
- 9个大模型在200个任务中仅3%的任务通过率超95%，无一完全解决任务
- 模型偏好单一文件的臃肿结构，与人类代码风格差异巨大
将想法转化为完整软件项目已成为语言模型的重要应用。智能体被用于在无需人工干预的情况下初始化、维护和扩展代码库。此类场景要求模型具备高层次的软件架构决策能力。然而现有基准仅评估修复单个缺陷或实现特定功能等局部任务。为此，我们提出ProgramBench，用于衡量软件工程智能体从零开始整体开发软件的能力。在ProgramBench中，给定一个程序及其文档，智能体需设计并实现一个与参考可执行文件行为一致的代码库。通过智能体驱动的模糊测试生成端到端行为测试，实现无需预设实现结构的评估。200个任务涵盖从简洁命令行工具到广泛使用的软件如FFmpeg、SQLite和PHP解释器。我们评估了9个大语言模型，发现没有一个能完全解决任何任务，最佳模型仅在3%的任务中通过95%的测试。模型倾向于采用单文件、全集成的实现方式，与人类编写的代码结构显著不同。
原文摘要 · Abstract (English)
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。