测试大模型在复杂依赖环境中的自动化能力,发现顶尖模型成功率仅60%。
ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox

- 构建动态交互工具沙箱,模拟真实软件环境的依赖与故障。
- 顶级模型在复杂任务中成功率不足60%,远低于人类的90%。
- 揭示三大瓶颈:工具检索饱和、过度自信、失败后放弃尝试。
当前大模型代理擅长调用孤立API,但在商业自动化‘最后一公里’仍显乏力。真实场景中,工具具有原子性、相互依赖性且易受环境噪声影响。我们提出$ extbf{ComplexMCP}$,一个基于模型上下文协议(MCP)的基准测试平台,包含来自7个有状态沙箱的300多个经严格验证的工具,涵盖办公套件至金融系统。不同于现有数据集,本基准采用种子驱动架构,模拟动态环境状态与不可预测的API故障,确保评估既确定又多样化。我们在全上下文与RAG范式下评估多种大模型,发现显著性能差距:即使顶尖模型成功率也未超过60%,远低于人类的90%。轨迹细粒度分析识别出三大根本瓶颈:(1) 工具检索饱和,随动作空间扩大而恶化;(2) 过度自信,跳过必要环境验证;(3) 战略性消极主义,倾向于为失败辩护而非尝试恢复。这些发现凸显现有代理在依赖工作流中的不足,确立$ extbf{ComplexMCP}$作为下一代韧性自主系统的关键测试平台。
原文摘要 · Abstract (English)
Current LLM agents are proficient at calling isolated APIs but struggle with the "last mile" of commercial software automation. In real-world scenarios, tools are not independent; they are atomic, interdependent, and prone to environmental noise. We introduce $\textbf{ComplexMCP}$, a benchmark designed to evaluate agents in these rigorous conditions. Built on the Model Context Protocol (MCP), $\textbf{ComplexMCP}$ provides over 300 meticulously tested tools derived from 7 stateful sandboxes, ranging from office suites to financial systems. Unlike existing datasets, our benchmark utilizes a seed-driven architecture to simulate dynamic environment states and unpredictable API failures, ensuring a deterministic yet diverse evaluation. We evaluate various LLMs across full-context and RAG paradigms, revealing a stark performance gap: even top-tier models fail to exceed a 60% success rate, far trailing human performance 90%. Granular trajectory analysis identifies three fundamental bottlenecks: (1) $\textbf{tool retrieval saturation}$ as action spaces scale; (2) $\textbf{over-confidence}$, where agents skip essential environment verifications; and (3) $\textbf{strategic defeatism}$, a tendency to rationalize failure rather than pursuing recovery. These findings underscore the insufficiency of current agents for interdependent workflows, positioning $\textbf{ComplexMCP}$ as a critical testbed for the next generation of resilient autonomous systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。