构建真实工具调用基准，评估大模型跨服务器协作能力。
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
- 设计1000个自然语言任务，需在36个真实MCP服务器中自主发现并组合工具。
- 采用原子事实评分机制，支持不同调用路径的正确性验证，通过率最高达82.2%。
- 提供诊断分类体系，发现63.3%失败源于理解或合成错误而非工具调用问题。
模型上下文协议（MCP）正成为大语言模型代理发现并调用外部工具的标准接口。然而现有MCP评估在三个关键维度存在不足：真实多步流程与跨服务器编排、真实MCP服务器覆盖而非模拟、可复现的逐条声明评分，且与代理表达冗余或风格解耦。我们提出MCP-Atlas，一个针对生产级MCP服务器的工具使用能力基准。该基准包含1000个由人类专家编写并验证的自然语言任务，覆盖36个真实MCP服务器和220个工具。提示不指定服务器、工具或参数，要求代理从语义上合理的干扰项中识别相关工具，并构建多步跨服务器工作流。每个任务采用声明级评分标准，最终答案依据工具输出中的原子事实进行评判。该以答为本的评分方式允许不同工具调用路径获得认可。我们还引入11类诊断分类体系，将工具调用失败与任务理解、综合、解析和终止等认知失败相分离。在六家厂商的20个前沿模型上进行匹配条件下的任务级评估，发现0.75声明覆盖率阈值下通过率最高达82.2%，并呈现明显的三层次性能结构。自动化诊断显示63.3%的失败属于认知层面。值得注意的是，部分高性能模型在成功执行工具后仍因过早停止或错误合成而失败。我们公开任务模板、容器化评测框架、声明评估器及500个任务的公共版本，保留500个任务私有版本以保障排行榜完整性。代码见https://github.com/scaleapi/mcp-atlas。
原文摘要 · Abstract (English)
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。