arXiv:2604.12896cs.CVcs.LG2026-04中稿 · CVPR被引 2

让大模型读懂视觉工具的提示,而非像素,提升视觉推理能力。

Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

论文配图:Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs
图 1 · 摘自论文原文
  • 将视觉工具输出转为语言可理解的结构化摘要,匹配大模型推理习惯。
  • 在多视角、相对深度等任务上,准确率最高提升45个百分点。
  • 无需训练或修改模型,适配各类大模型,尤其对小模型效果显著。

多模态语言模型(MLLMs)常搭配深度、光流等视觉工具以增强视觉推理能力,但现有方法直接输入原始工具输出,其密集的像素级信息与语言模型的推理方式不匹配,导致感知能力弱、依赖语言先验。本文提出感知程序(P²),一种无需训练、模型无关的方法,将工具输出重写为紧凑、结构化、语言原生的摘要,使MLLM可直接解析与推理。在BLINK数据集的六项感知任务中,P²显著优于基础模型和原始工具增强基线。以GPT-5 Mini为基底,多视角推理准确率从41.35%提升至86.47%,相对深度任务从52.42%升至81.45%,平均提升22%,达新最优。即使在InternVL3.5-4B、Qwen3VL-4B等小型模型上,仍实现15%-40%的绝对提升,超越此前所有代理式、监督及强化学习类工具使用方法,且无需任何训练或模型修改。

原文摘要 · Abstract (English)

Multimodal language models (MLLMs) are increasingly paired with vision tools (e.g., depth, flow, correspondence) to enhance visual reasoning. However, despite access to these tool-generated visual cues, MLLMs often fail to benefit from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, leading to weak perception and reliance on language priors. We argue that, in problems where vision tools can provide the necessary visual cues, the bottleneck is not more tool calls or larger MLLMs, it is how tool outputs are represented. We introduce Perception Programs (P$^2$), a training-free, model-agnostic method that rewrites tool outputs into compact, structured, language-native summaries that MLLMs can directly parse and reason over. Across six perception-centric tasks in BLINK, P$^2$ consistently yields large improvements over base models and raw tool-augmented baselines. With GPT-5 Mini as the base model, P$^2$ raises its accuracy from 41.35\% to 86.47\% on multi-view reasoning, from 52.42\% to 81.45\% on relative depth, and achieves a 22\% average gain across tasks, setting new state-of-the-art results. Even on smaller MLLMs, e.g., InternVL3.5-4B and Qwen3VL-4B, we observe 15-40\% absolute gains from P$^2$, surpassing prior agentic, supervised, and RL-based tool-use methods-without any training or model modifications.

视觉推理感知程序多模态工具使用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。