让AI看图时能长期用工具，解决视觉理解变差问题
ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use
- 分三步：先规划全局，再局部调用搜索/代码/看图工具，最后整合答案
- 在4个测试集上平均提升6.69%，长序列任务表现尤其好
- 适合需要持续看图并用工具推理的复杂任务，如科学问答
大语言模型通过调用外部工具实现自主解题能力突出，但多模态大模型在复杂多模态信息下灵活高效使用工具仍面临挑战。本文提出ToolScope，一种统一全局规划与局部多模态感知的智能体框架。该框架包含三个核心组件：全局导航器（作为‘望远镜’提供高层策略）、智能执行器（迭代调用外部工具——搜索、代码、感知）以及响应合成器。其中，专用感知工具有效缓解长时程视觉问答任务中的视觉上下文退化问题。我们在VQA 2.0、ScienceQA、MAT-Search和MathVista四个跨领域基准上评估，结果显示ToolScope具备强大泛化能力，所有数据集平均性能提升达+6.69%。
原文摘要 · Abstract (English)
Recently, large language models (LLMs) have demonstrated remarkable problem-solving capabilities by autonomously integrating with external tools for collaborative reasoning. However, due to the inherently complex and diverse nature of multimodal information, enabling multimodal large language models (MLLMs) to flexibly and efficiently utilize external tools during reasoning remains an underexplored challenge. In this work, we introduce ToolScope, an agentic framework designed to unify global planning with local multimodal perception, adopting a specialized Perceive tool to mitigates visual context degradation in long-horizon VQA task. ToolScope comprises three primary components: the Global Navigator, the Agentic Executor, and the Response Synthesizer. The Global Navigator functions as a "telescope", offering high-level strategic guidance. The Agentic Executor operates iteratively to augment MLLM with local perception through the integration of external tools-Search, Code, and Perceive. Finally, the Response Synthesizer consolidates and organizes the reasoning process into a coherent, user-friendly output. We evaluate ToolScope on four VQA benchmarks across diverse domains, including VQA 2.0, ScienceQA, MAT-Search and MathVista. It demonstrates strong generalization capabilities, achieving an average performance improvement of up to +6.69% across all datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。