arXiv:2603.29902cs.AI2026-03中稿 · ECCV

让大模型像指挥家一样协调工具生成图文混排内容。

ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation

  • 设计多智能体系统评估模型自主规划工具调用的能力。
  • 在7702个问答对上测试,发现现有模型在图文交替生成中表现不佳。
  • 适合关注多模态推理与智能体规划的研究者和开发者。

图文交替生成是多模态大模型(MLLM)的重要前沿方向,能更直观地传递复杂信息。当前方法或依赖图像生成或检索增强,但通常将二者视为互斥路径,难以兼顾事实性与创造性。我们提出,该领域下一里程碑应为「代理工具规划」:模型作为核心控制器,自主决定何时、何地、调用何种工具,以生成针对视觉关键问题的交错响应。为此,我们构建了ATP-Bench基准,包含7702个问答对(含1592个视觉问答对),覆盖8个类别和25种视觉关键意图,所有查询与答案均经人工验证。为进一步独立评估工具规划能力,不依赖端到端执行或工具后端变化,我们提出多智能体判官系统(MAM),可评估工具调用精度、识别遗漏的工具使用机会,并评估整体响应质量,无需真实参考答案。在10个先进MLLM上的大量实验表明,模型在连贯的图文交替规划上存在明显短板,工具使用行为差异显著,揭示巨大提升空间,并为推进图文交错生成提供可操作指导。数据集与代码已开源:https://github.com/Qwen-Applications/ATP-Bench。

原文摘要 · Abstract (English)

Interleaved text-and-image generation represents a significant frontier for Multimodal Large Language Models (MLLMs), offering a more intuitive way to convey complex information. Current paradigms rely on either image generation or retrieval augmentation, yet they typically treat the two as mutually exclusive paths, failing to unify factuality with creativity. We argue that the next milestone in this field is Agentic Tool Planning, where the model serves as a central controller that autonomously determines when, where, and which tools to invoke to produce interleaved responses for visual-critical queries. To systematically evaluate this paradigm, we introduce ATP-Bench, a novel benchmark comprising 7,702 QA pairs (including 1,592 VQA pairs) across eight categories and 25 visual-critical intents, featuring human-verified queries and ground truths. Furthermore, to evaluate agentic planning independent of end-to-end execution and changing tool backends, we propose a Multi-Agent MLLM-as-a-Judge (MAM) system. MAM evaluates tool-call precision, identifies missed opportunities for tool use, and assesses overall response quality without requiring ground-truth references. Our extensive experiments on 10 state-of-the-art MLLMs reveal that models struggle with coherent interleaved planning and exhibit significant variations in tool-use behavior, highlighting substantial room for improvement and providing actionable guidance for advancing interleaved generation. Dataset and code are available at https://github.com/Qwen-Applications/ATP-Bench.

多模态智能体规划图文生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。