评测大模型在复杂企业流程中的执行能力，发现普遍难以应对多分支逻辑与对话细节。
SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures
- 构建397个实例、3422个子任务的业务流程基准，分横向与纵向两类任务
- 多数顶尖模型在流程追踪、对话理解、时间计算上错误率超60%
- 揭示流程盲视、对话脆弱性、计算错误三大短板，适合评估企业级Agent
随着大语言模型（LLMs）被广泛部署为领域专用智能体，已有诸多基准用于评估其在真实场景中遵循指令与决策的能力。然而，企业场景常涉及复杂的标准操作流程（SOP），而针对此类情境下LLM能力的评估尚未充分展开。为此，我们提出SOP-Maze，基于真实业务数据构建，涵盖23个复杂SOP场景，包含397个实例和3422个子任务。我们将SOP任务分为两类：横向根系（LRS），代表需精准选择的宽选项任务；纵向根系（HRS），强调深层逻辑推理与复杂分支判断。大量实验表明，几乎所有先进模型在SOP-Maze上表现不佳。通过系统分析，我们识别出三类关键错误：(i) 路径盲视——难以追踪流程步骤；(ii) 对话脆弱性——无法处理真实对话中的细微变化；(iii) 计算错误——在复杂上下文中出现时间或算术推理失误。本研究深入探讨了模型在广度与深度双重挑战下的表现，为提升模型能力提供了新洞见。相关代码与数据已开源：https://github.com/meituan-longcat/SOP-Maze。
原文摘要 · Abstract (English)
As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on: https://github.com/meituan-longcat/SOP-Maze.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。