模型推理的关键不是图像,而是调用工具时生成的文本指令。
Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

- 用文本替代返回图像,仅保留工具调用的结构化文字信息
- 新方法在多项测试中保持甚至超越原模型准确率
- 适合追求低延迟和减少接口调用的视觉推理应用
增强工具的视觉语言模型越来越依赖图像进行推理:它们调用裁剪、缩放或代码工具,并基于返回的像素进行思考。然而,近期通过盲测、增益分解和注意力分析发现,返回的图像贡献极小。本文提出疑问:若像素不承载主要信息,那真正起作用的是什么?我们假设核心信号是工具调用前生成的结构化文本——包括工具名称、坐标、目标描述和意图。该文本指引观察位置与目标。为此,我们提出TextCall(无图像返回):保留文本指令但将返回图像替换为[Image output skipped]。三项研究支持该假设:(i) 返回图像非必需:在LoRA、全微调和强化学习设置下,TextCall表现相当或更优;强化学习中,它维持工具调用行为,避免了因看到图像后直接回答而导致的失败模式。(ii) 文本支架已足够:在匹配训练查询上,仅用文本输入可达到与含图像输入相同的准确率。(iii) 组分特异性:分解文本支架发现,推理文本与空间编码均有效,主导成分随任务变化。结果支持‘工具调用文本支架假说’:当前‘以图像思考’范式中,活跃信号是工具调用时刻产生的结构化文本;返回图像仅为冗余载体。TextCall在保持准确率的同时,降低29%-46%延迟并消除工具执行API调用。此结论适用于现有‘以图像思考’基准;构建真正依赖像素的任务仍是开放问题。
原文摘要 · Abstract (English)
Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding the failure mode where, under matched settings, seeing the returned image causes the model to stop calling tools and answer directly. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Component specificity: decomposing the scaffold into reasoning text and spatial code shows both components contribute, with the dominant one varying by task. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29-46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。