测试大模型在企业复杂系统中的真实表现，发现其难以预判隐藏流程的连锁反应。
World of Workflows: A Benchmark for Bringing World Models to Enterprise Systems
- 构建基于ServiceNow的真实企业环境，含4000+规则与55条隐性工作流
- 前沿大模型在234项任务中频繁因无法预见连锁副作用而失败
- 强调需建立对系统动态的具身化认知，才能应对不可见状态变化
前沿大语言模型（LLMs）在诸多领域作为自主代理表现出色，但在复杂的企事业单位系统中仍缺乏验证。这些系统中隐藏的工作流会在相互关联的数据库间引发级联效应。现有企业基准评估的仅是表层的代理任务完成度，类似于通用消费类基准，忽略了企业中的真实挑战：可观测性受限、数据库状态庞大，以及具有级联副作用的隐藏工作流。为此，我们提出了「世界工作流」（World of Workflows, WoW），一个基于ServiceNow的逼真环境，内嵌4000多个业务规则和55条活跃工作流，并配套推出WoW-bench基准，包含234个任务，用于评估代理在约束条件下的任务完成能力及对企业动态建模的能力。我们得出两个关键结论：(1) 前沿大模型存在‘动态盲视’问题，持续无法预测其操作带来的不可见级联副作用，导致沉默的约束违规；(2) 在透明度低的系统中实现可靠性，需要基于世界的建模能力，即代理必须在头脑中模拟隐藏的状态转移，以弥补高保真反馈缺失带来的可观测性鸿沟。为实现可靠且实用的企业代理，WoW推动了一种新范式——显式学习系统动态。我们已将代码开源，供部署与评估。
原文摘要 · Abstract (English)
Frontier large language models (LLMs) excel as autonomous agents in many domains, yet they remain untested in complex enterprise systems where hidden workflows create cascading effects across interconnected databases. Existing enterprise benchmarks evaluate surface-level agentic task completion similar to general consumer benchmarks, ignoring true challenges in enterprises, such as limited observability, large database state, and hidden workflows with cascading side effects. We introduce World of Workflows (WoW), a realistic ServiceNow-based environment incorporating 4,000+ business rules and 55 active workflows embedded in the system, alongside WoW-bench, a benchmark of 234 tasks evaluating constrained agentic task completion and enterprise dynamics modeling capabilities. We reveal two major takeaways: (1) Frontier LLMs suffer from dynamics blindness, consistently failing to predict the invisible, cascading side effects of their actions, which leads to silent constraint violations, and (2) reliability in opaque systems requires grounded world modeling, where agents must mentally simulate hidden state transitions to bridge the observability gap when high-fidelity feedback is unavailable. For reliable and useful enterprise agents, WoW motivates a new paradigm to explicitly learn system dynamics. We release our GitHub for setting up and evaluating WoW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。