构建视觉工具调用评估框架，揭示大模型视觉感知短板。
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
- 设计支持500+工具的动态执行环境，模拟多轮视觉交互任务。
- 评测12个主流模型，最佳仅达49.7%成功率，视觉理解是主要瓶颈。
- 揭示模型规模越大，越易因看不清图像出错，适合作为评测基准。
我们提出MM-ToolSandBox，一个面向视觉引导工具调用智能体的基准与评估框架。该框架提供跨越16个应用领域的500多个工具状态化执行环境，支持多图像、多轮次任务，要求智能体将逐步输入的视觉信息准确映射为可执行工具调用，并处理真实对话中的目标修改、纠错与状态变更。通过信息流引导规划和多阶段质量过滤的自动化场景生成管道，共生成258个经人工验证的标准场景及50个针对交互式UI的应用变体。对12个前沿模型（包括4B开源与顶尖闭源系统）的评估显示，当前模型在视觉工具调用上仍缺乏鲁棒性：即使最优模型成功率也低于50%。失败分析表明，53%的错误源于图像信息提取不准确，而非任务规划失误。随着模型规模增长，问题从“不知做什么”转向“看不清做什么”，凸显不同能力层级需采取差异化改进路径。框架与基准已开源。
原文摘要 · Abstract (English)
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。