测试大模型在真实软件服务中完成专业任务的能力。
SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?
- 构建23个真实SaaS系统上的106个专业任务，覆盖长周期工作流。
- 最强模型仅完成不足4%的任务，暴露出规划与状态追踪短板。
- 适合研究智能体在复杂真实场景中的应用与局限。
计算机使用智能体（CUAs）正将大语言模型（LLMs）从文本推理拓展至网页和图形界面等复杂环境中的动作执行。然而，现有基准多依赖简化场景、孤立任务或短时交互，难以评估智能体在真实专业工作流中的表现。软件即服务（SaaS）环境天然适合作为评估场景，因其承载现代数字工作的主要部分，包含动态系统状态、跨应用协作、领域知识及长周期依赖。为此，我们提出SaaS-Bench，基于六个专业领域中的23个可部署SaaS系统，构建了106个源于真实工作场景的任务。这些任务需长周期执行，涵盖纯文本与多模态设置，并采用加权验证节点评估任务完成度与部分进展。实验表明，代表性基于LLM的智能体在SaaS-Bench上表现不佳，即使最强模型也仅能端到端完成少于4%的任务，暴露出规划、状态追踪、跨应用上下文保持及错误恢复等方面的严重不足。代码已开源：https://github.com/UniPat-AI/SaaS-Bench。
原文摘要 · Abstract (English)
Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments, such as web browsers and graphical user interfaces (GUIs). However, existing web and GUI agent benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions, making it difficult to assess capabilities of agents in realistic professional workflows. Software-as-a-Service (SaaS) environments are a natural choice for CUA evaluation, as they host a large share of modern digital work and naturally involve dynamic system states, cross-application coordination, domain-specific knowledge, and long-horizon dependencies. To this end, we introduce SaaS-Bench, a benchmark built on 23 deployable SaaS systems across six professional domains, containing 106 tasks grounded in realistic work scenarios. These tasks require long-horizon execution, cover both text-only and multimodal settings, and are evaluated with weighted verification checkpoints that measure strict task completion and partial progress. Experiments show that representative LLM-based agents struggle on SaaS-Bench, with even the strongest model completing fewer than 4% of tasks end-to-end, exposing limitations in planning, state tracking, cross-application context maintenance, and error recovery. Code are available at https://github.com/UniPat-AI/SaaS-Bench for reproduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。