arXiv:2506.08119cs.AI2025-06KDD被引 19

评测大模型在复杂工业流程中的执行能力,提供真实场景的基准测试集。

SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents

  • 基于专家撰写的真实工业操作流程,构建2000+任务的评估基准。
  • 不同模型表现差异大,新模型未必更优,如Claude 4.5在部分任务上低于4.0。
  • 适合研究者测试模型设计、工具调用和部署策略,避免真实场景试错成本。

基于大模型的智能体难以执行复杂的多步骤标准操作流程(SOP),而现有基准无法反映真实工作流的流程复杂性和工具协调需求。我们提出SOP-Bench,包含来自12个业务领域(医疗、物流、金融、内容审核等)的2000多个由人类专家撰写的SOP任务。通过人机协作框架,专家制定真实流程,AI生成工具、API和数据集,并经人工验证,确保任务具备可执行接口与真实输出。该基准支持系统性研究智能体架构、模型能力与部署策略。我们使用前沿模型对部分任务进行实验,发现:(1)新模型不保证更好性能——Claude 4 Opus在ReAct任务中成功率72.4%,高于Claude 4.5 Sonnet的63.3%;(2)无单一最优组合,最佳表现从57%到100%不等,依领域而异。这些结果表明SOP-Bench可有效分离并研究智能体性能的关键维度,无需昂贵的生产级实验。目标不是排名模型或打造最优代理,而是提供严谨的评估框架,供研究者与从业者系统分析代理设计、模型选择与部署方案。基准已开源:https://github.com/amazon-science/sop-bench。

原文摘要 · Abstract (English)

LLM-based agents struggle to execute complex, multi-step Standard Operating Procedures (SOPs) that are fundamental to industrial automation. Existing benchmarks fail to capture the procedural complexity and tool orchestration demands of real-world workflows. We introduce SOP-Bench, a benchmark of 2,000+ tasks from human expert-authored SOPs across 12 business domains (healthcare, logistics, finance, content moderation, etc.). Using a human-AI collaborative framework, experts crafted authentic SOPs while AI generated artifacts (tools, APIs, datasets), all human-validated, yielding realistic tasks with executable interfaces and ground-truth outputs. SOP-Bench serves as a research enabler for systematically investigating agent architectures, model capabilities, and deployment considerations across diverse procedural tasks. We demonstrate its utility through illustrative experiments with a subset of frontier models across Function-Calling (FC) and ReAct agents, revealing critical insights. For example, (1) newer models do not guarantee better performance - Claude 4 family outperforms Claude 4.5 family on ReAct tasks (Claude 4 Opus: 72.4% vs. Claude 4.5 Sonnet: 63.3% task success rate), demonstrating that production upgrades require validation; (2) no single model-agent combination dominates: best performances range from 57% to 100% depending on domain. These examples illustrate how SOP-Bench enables isolating and studying specific dimensions of agent performance without costly production experiments. Our goal is not to rank model capabilities or build optimal agents, but to provide a rigorous evaluation framework that enables the researchers and practitioners to systematically investigate agent design choices, model selection, and deployment strategies. We release the benchmark at https://github.com/amazon-science/sop-bench.

大模型评估工业自动化SOP评测智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。