模型自适应决定是否调用工具，提升多模态推理准确率与效率
Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning
- 基于强化学习设计双模式推理机制，动态判断是否调用工具
- 在V*上比基线模型高21.8%准确率，在POPE上效率提升44.9%
- 适合需要高效精准多模态推理的场景，如智能助手、视觉问答
工具增强推理已成为提升多模态大模型推理能力的有前景方向。然而，现有研究主要关注模型能否调用工具，忽视了调用工具的必要性。我们指出，工具使用并非总是有益，冗余或不当调用会大幅增加推理开销，甚至误导预测结果。为此，我们提出AutoTool，一种根据查询特征自适应决定是否调用工具的模型。在强化学习框架下，设计了具有模式特异性奖励函数的双模式推理策略，引导模型生成准确回答。此外，为防止过早偏向单一推理模式，AutoTool在训练中联合探索并平衡工具辅助与文本中心推理，并在后期促进自由探索。大量实验表明，AutoTool表现优异且高效：在V*基准上相比基线模型提升21.8%准确率，在POPE基准上效率优于现有方法44.9%。代码已开源。
原文摘要 · Abstract (English)
Tool-augmented reasoning has emerged as a promising direction for enhancing the reasoning capabilities of multimodal large language models (MLLMs). However, existing studies mainly focus on enabling models to perform tool invocation, while neglecting the necessity of invoking tools. We argue that tool usage is not always beneficial, as redundant or inappropriate invocations largely increase reasoning overhead and even mislead model predictions. To address this issue, we introduce AutoTool, a model that adaptively decides whether to invoke tools according to the characteristics of each query. Within a reinforcement learning framework, we design an explicit dual-mode reasoning strategy with mode-specific reward functions to guide the model toward producing accurate responses. Moreover, to prevent premature bias toward a single reasoning mode, AutoTool jointly explores and balances tool-assisted and text-centric reasoning throughout training, and promotes free exploration in later stages. Extensive experiments demonstrate that AutoTool exhibits outstanding performance and high efficiency, yielding a 21.8\% accuracy gain on V* benchmark compared to the base model, and a 44.9\% improvement in efficiency over existing tool-augmented methods on POPE benchmark. Code is available at https://github.com/MQinghe/AutoTool.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。