让AI智能判断何时用视觉工具，减少无用计算提升效率
AdaTooler-V: Adaptive Tool-Use for Images and Videos
- 根据工具收益动态调整奖励，只在必要时调用视觉工具
- 在12个基准上表现优异，7B模型在高分辨率任务达89.8%准确率
- 适合需要高效视觉推理的开发者和研究者
近期研究表明，多模态大语言模型（MLLM）通过与视觉工具交互的多模态链式思维（CoT）可提升性能。然而，现有开源模型常存在盲目调用工具的问题，即使无需工具也频繁使用，显著增加推理开销并降低性能。为此，我们提出AdaTooler-V，一种能自适应判断是否需要工具的MLLM。首先，引入AT-GRPO强化学习算法，依据每样本的工具收益评分动态调整奖励尺度，促使模型仅在工具带来实际改进时才调用。此外，构建两个数据集支持训练：用于监督微调的AdaTooler-V-CoT-100k，以及用于强化学习的AdaTooler-V-300k，覆盖单图、多图和视频数据。在十二个基准上的实验表明，AdaTooler-V具备强大推理能力，其7B版本在高分辨率基准V*上达到89.8%准确率，优于GPT-4o和Gemini 1.5 Pro。所有代码、模型与数据均已公开。
原文摘要 · Abstract (English)
Recent advances have shown that multimodal large language models (MLLMs) benefit from multimodal interleaved chain-of-thought (CoT) with vision tool interactions. However, existing open-source models often exhibit blind tool-use reasoning patterns, invoking vision tools even when they are unnecessary, which significantly increases inference overhead and degrades model performance. To this end, we propose AdaTooler-V, an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. First, we introduce AT-GRPO, a reinforcement learning algorithm that adaptively adjusts reward scales based on the Tool Benefit Score of each sample, encouraging the model to invoke tools only when they provide genuine improvements. Moreover, we construct two datasets to support training: AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL with verifiable rewards across single-image, multi-image, and video data. Experiments across twelve benchmarks demonstrate the strong reasoning capability of AdaTooler-V, outperforming existing methods in diverse visual reasoning tasks. Notably, AdaTooler-V-7B achieves an accuracy of 89.8\% on the high-resolution benchmark V*, surpassing the commercial proprietary model GPT-4o and Gemini 1.5 Pro. All code, models, and data are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。