让视觉语言模型更聪明地决定何时调用外部工具，省 token 又提准确率。
ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents
- 用轨迹文本和结构特征预测是否执行工具调用，轻量高效。
- 在五个基准上降低64%-69%的token消耗，跨域准确率基本不变。
- 特定领域训练后准确率提升1.65点，适合资源受限的多模态应用。
增强视觉语言模型通过OCR、检测、分割等工具获取外部感知证据，但每次工具调用成本高且未必必要。我们研究预调用控制问题：在ReAct风格的VLM提出感知工具调用后，是否应执行？在五个基准上，基线模型表现出差的局部选择性：有益与有害调用发生率相近（11.8% vs. 9.9%），多数调用不影响即时强制答案预测。我们提出ToolGate，一种轻量级外部控制器，基于轨迹文本和简单结构特征预测执行/跳过决策。在两个Qwen3-VL骨干模型上，ToolGate将token成本降至无约束ReAct基线的64%-69%，同时在跨域设置中保持平均准确率。在匹配领域轨迹训练下，其平均准确率进一步提升1.65点。结果表明，工具增强型VLM不仅受益于更好的感知工具，也受益于对工具输出价值的显式控制。
原文摘要 · Abstract (English)
Tool-augmented vision-language agents can acquire external perceptual evidence through OCR, detection, segmentation, and other tools, but executing every proposed tool call is costly and sometimes unnecessary. We study the pre-call control problem: after a ReAct-style VLM agent proposes a perceptual tool call, should the call be executed, or skipped before its output enters the context? Across five benchmarks, we find that the baseline agent exhibits poor local selectivity: helpful and harmful calls occur at similar rates (11.8% vs. 9.9%), while most calls do not change the immediate forced-answer prediction. We introduce ToolGate, a lightweight external controller that predicts execute/skip decisions from trajectory text and simple structural features. Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline while preserving average accuracy in cross-domain settings. With matched-domain trajectory training on Qwen3-VL-30B, it further improves average accuracy by 1.65 points. These results show that tool-augmented VLM agents benefit not only from better perceptual tools, but also from explicit control over when tool outputs are worth paying for.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。