arXiv:2605.17526cs.SEcs.AI2026-05被引 2

首个面向企业级SaaS系统的编码智能体评测基准,揭示系统集成才是关键瓶颈。

SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

论文配图:SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
图 1 · 摘自论文原文
  • 构建覆盖6大SaaS领域、30个复杂任务的多语言多框架评测体系
  • 95%以上失败源于系统配置而非业务逻辑,暴露集成能力短板
  • 适合研究系统级智能体、长周期开发的开发者与研究者使用

随着自主编码智能体逐步承担更长周期的任务,其在端到端软件开发中的潜力日益显现。然而现有评测基准仍局限于结构简单、单技术栈的应用,难以反映真实企业SaaS系统在异构环境、全栈协同与系统复杂性方面的挑战。为此,我们提出SaaSBench,首个面向企业SaaS工程的智能体评测基准。该基准涵盖6个SaaS领域、30个复杂任务,包含5,370个验证节点,支持8种编程语言、6种数据库和13个框架,高度还原真实软件多样性。我们设计了依赖感知的混合评估范式,适用于长周期、多组件耦合的复杂系统,实现细粒度可复现的评估。大量实验揭示关键发现:当前先进智能体的主要瓶颈并非生成孤立代码逻辑,而是成功配置并集成多组件系统。超过95%的任务失败发生在深入业务逻辑之前,模型常因过度自信过早终止基础系统搭建,或陷入无效调试循环。我们希望SaaSBench能成为推动可靠系统级编码智能体发展的实用且具挑战性的测试平台。代码已开源。

原文摘要 · Abstract (English)

As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95\% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents. The code is available at \url{https://github.com/ShadeCloak/SaaSbench}.

编码智能体SaaS评测系统集成长周期任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。