arXiv:2508.01780cs.AIcs.CL2025-08KDD被引 51

评测大模型在万级工具中的实时导航能力,发现检索失败是主要瓶颈。

LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?

  • 构建70个服务器527个工具的真实任务集,模拟多服务器复杂路由。
  • 12个模型平均成功率仅30-50%,最强模型为Claude-Sonnet-4的78.95%。
  • 首次揭示工具组合能力与检索错误对性能的关键影响,适合评估智能体系统。

Model Context Protocol (MCP) 已成为连接大模型与外部工具的关键基础设施,已扩展至超过10,000个服务器和多样化的工具。然而,现有评估方法仍存在巨大差距:通常假设单服务器场景,直接将工具注入模型上下文,忽视了大规模检索与多工具组合的实际挑战。为此,我们提出LiveMCPBench,用于评估95项真实世界日常任务,明确设计以考验多样化工具与规模化多服务器路由。该基准包含一个可直接部署的工具套件,涵盖70个服务器和527个工具,确保复现性且无需分散配置API。我们进一步引入基于大模型作为裁判(LLM-as-a-Judge)的评估框架,可直接验证任务结果,支持动态数据源与多种有效解决方案路径。我们在12个先进大模型上进行测试,发现显著性能差距:Claude-Sonnet-4达到78.95%任务成功率,而多数模型仅为30-50%。分析表明,主动工具组合与任务成功强相关,而检索错误占所有失败的近一半,凸显检索是主要瓶颈。这些结果首次提供了大规模、可复现的MCP智能体能力诊断,并指向未来研究方向:提升检索鲁棒性与促进有效工具组合。代码与数据公开于https://icip-cas.github.io/LiveMCPBench。

原文摘要 · Abstract (English)

Model Context Protocol (MCP) has become a key infrastructure for connecting LLMs with external tools, scaling to 10,000+ MCP servers with diverse tools. Unfortunately, there is still a large gap between real-world MCP usage and current evaluation: they typically assume single-server settings and directly inject tools into the model's context, bypassing the challenges of large-scale retrieval and multi-tool composition. To bridge this gap, we propose LiveMCPBench, which evaluates 95 real-world daily tasks explicitly constructed to stress diverse tools and scaled multi-server routing. The benchmark includes a ready-to-deploy tool suite of 70 servers with 527 tools, ensuring reproducibility without scattered API configuration. We further introduce an LLM-as-a-Judge evaluation framework that directly verifies task outcomes, handling dynamic data sources and multiple valid solution paths. We benchmark 12 state-of-the-art LLMs and observe a substantial performance gap: while Claude-Sonnet-4 reaches 78.95% task success, most models achieve only 30-50%. Our analysis reveals that the active tool composition strongly correlates with task success, whereas retrieval errors account for nearly half of all failures, highlighting retrieval as the dominant bottleneck. Together, these results provide the first large-scale, reproducible diagnosis of MCP agent capabilities and point towards future research on improving retrieval robustness and encouraging effective tool composition. Our code and data are publicly available at https://icip-cas.github.io/LiveMCPBench.

智能体评估工具调用大模型测评

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。