测试大模型从零生成可运行命令行工具的能力，发现顶尖模型成功率仅43.8%。
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
- 无模板、无结构，要求模型从空目录自主搭建完整工具。
- 用黑盒差分测试验证输出是否与人工代码一致，成功率最高43.8%。
- 揭示模型倾向写大块代码，且耗更多词不等于表现更好。
大型语言模型的演进推动了以意图驱动的软件开发范式，即自主智能体需从零开始设计并交付完整可运行的软件系统。然而现有基准存在两大缺陷：其一，依赖预设结构模板，使任务退化为文件填充；其二，采用严格的白盒单元测试，强制代码符合特定内部实现，而非验证端到端用户行为。为此，我们提出CLI-Tool-Bench，一个结构无关的新基准，用于评估命令行工具的从零生成能力。该基准基于自动化黑盒差分测试框架，包含94个高质量真实仓库，覆盖多种编程语言和复杂度。每个任务中，智能体在空工作区启动，需自主规划项目结构与依赖。生成代码在隔离沙箱中执行，系统级副作用与终端输出通过多层级等价度量与人工基线比对。对七种先进LLM的评估显示，顶级模型最高整体成功率仅为43.8%，表明0到1软件生成仍是严峻挑战。此外，我们发现智能体普遍生成单体代码结构，且更高词数消耗并不保证更好性能。
原文摘要 · Abstract (English)
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch. However, existing benchmarks fail to adequately assess this 0-to-1 generation capability due to two fundamental limitations. First, they rely on predefined structural scaffolds, which reduces the task to mere file-filling. Second, they depend on rigid white-box unit testing, which forces generated code to conform to specific internal implementations rather than validating end-to-end user-centric behavior. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by an automated black-box differential testing framework, the benchmark comprises 94 high-quality, real-world repositories spanning diverse programming languages and complexity levels. For each task, agents are provided with an empty workspace, forcing them to autonomously handle repository planning and dependencies. We evaluate the generated software by executing it in isolated sandboxes. The system-level side effects and terminal outputs are then compared against human-written oracles using a rigorous multi-tiered equivalence metric. Extensive evaluation of seven state-of-the-art LLMs reveals that the top-tier models achieve a maximum overall success rate of only 43.8%, highlighting that 0-to-1 software generation remains a highly challenging frontier. Furthermore, we discover that agents exhibit a strong tendency to generate monolithic code structures, and that higher token consumption does not necessarily yield better task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。