测试大模型代码代理在零基础环境下的搭建能力，发现其实际表现远低于预期。
SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
- 从空Linux环境开始，自动安装依赖、配置数据库和服务
- 代理在项目启动和本地数据库配置上成功率仅20%-57%
- 适合关注真实开发场景下代理实用性的研究者和开发者
当前大型语言模型（LLM）代理虽承诺提供端到端的软件开发支持，但现有评测大多在预配置环境中进行。为填补这一空白，我们提出SetupBench，一个包含93个实例的基准测试，专门评估代理从裸机Linux沙箱中自主搭建开发环境的能力：包括安装包、解决依赖冲突、初始化数据库及配置后台服务。任务覆盖七种语言生态、五种数据库引擎和多服务编排场景，每个任务配有自然语言描述和确定性成功指令。对先进代码代理OpenHands的评估显示，各类任务成功率普遍偏低，项目搭建成功率38.9%-57.4%，本地数据库配置成功率20.0%-53.3%。分析揭示系统性失败模式：开发工具安装不全、任务约束幻觉、环境修改不可持久，导致人机协作流程中断。代理探索策略效率低下，38%-89%的操作为冗余行为。这些结果凸显当前代理在真实环境搭建能力上的显著不足。SetupBench为下一代面向端到端真实任务的开发者代理提供了严格评估标准。
原文摘要 · Abstract (English)
Modern Large Language Model (LLM) agents promise end to end assistance with real-world software tasks, yet existing benchmarks evaluate LLM agents almost exclusively in pre-baked environments where every dependency is pre-installed. To fill this gap, we introduce SetupBench, a 93 instance benchmark that isolates the environment-bootstrap skill: starting from a bare Linux sandbox, an agent must install packages, resolve dependency conflicts, initialize databases, and configure background services. Our tasks span seven language ecosystems, five database engines, and multi-service orchestration scenarios, each accompanies by a natural language problem statement and a deterministic success command. Through evaluation of OpenHands, a state-of-the-art coding agent, we find low success rates across task categories, with particular challenges in repository setup (38.9-57.4%) and local database configuration (20.0-53.3%). Our analysis reveals systematic failure modes including incomplete development tooling installation, hallucinated task constraints, and non-persistent environment modifications that break agent-human collaboration workflows. We identify substantial inefficiencies in agent exploration strategies, with 38-89% of actions being unnecessary compared to optimal human behavior. These findings highlight gaps in current agents' practical environment-bootstrap capabilities. By targeting this critical yet under-evaluated capability, SetupBench provides a rigorous yard-stick for the next generation of software developer agents aiming to solve end to end real-wold tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。