首个评估多模态工具使用的基准，聚焦复杂任务流程的准确性与一致性。
M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
- 通过语义相似度对齐工具调用，实现可审计的一一对应
- 28个服务器、231个工具，评测主流多模态大模型在跨工具任务中的差距
- 适合研究多模态推理与工具使用协同机制的研究者
我们提出M^3-Bench，首个基于模型上下文协议（MCP）评估多模态工具使用的基准。该基准针对需要视觉定位、文本推理、跨工具依赖和中间资源持久化的现实复杂工作流。引入基于相似度的对齐方法：序列化每个工具调用，用句向量编码签名，通过相似度分桶的匈牙利匹配获取可审计的一一对应关系。在此基础上报告解耦语义保真度与流程一致性的可解释指标。基准涵盖28个服务器上的231个工具，轨迹经执行器与裁判管道及人工验证标准化；额外由4个大型语言模型组成的评判集合报告最终任务完成率与信息锚定性。对代表性先进多模态大模型（MLLMs）的评估揭示其在多模态工具使用中仍存在显著缺陷，尤其在参数保真度与结构一致性方面，凸显需联合推理图像、文本与工具图的方法需求。基准匿名仓库地址：https://github.com/EtaYang10th/Open-M3-Bench
原文摘要 · Abstract (English)
We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。