让视觉语言模型学会看少算多,按需调用像素级分析
Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
- 根据问题难度动态决定是否启用像素级推理
- 在4K高分辨率任务上达73.4%准确率,工具使用率仅20.1%
- 适合需要精准视觉理解的复杂多模态任务
视觉语言模型在多模态任务中表现优异,但在需要精细理解细粒度视觉元素的任务上常遇瓶颈,主要源于图像编码过程中的信息丢失或对关键区域关注不足。近期研究通过将像素级视觉信息引入推理过程,使模型能在思考中访问高分辨率细节。然而,这类信息常被过度使用,导致效率低下并分散注意力。为此,我们提出首个自适应像素推理框架,基于输入查询动态决定是否执行像素级操作。首先通过操作感知的监督微调建立文本推理与视觉操作基础能力,再设计一种依赖模型自身回答反馈的滚动引导强化学习框架,使模型能根据问题难度自主判断何时调用像素操作。在多个多模态推理基准上的实验表明,该模型在显著减少不必要的视觉操作的同时取得更优性能。令人印象深刻的是,在HR-Bench 4K上达到73.4%准确率,工具使用率仅为20.1%,相比先前方法准确率提升且工具使用减少66.5%。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at many multimodal tasks, yet they frequently struggle with tasks requiring precise understanding and handling of fine-grained visual elements. This is mainly due to information loss during image encoding or insufficient attention to critical regions. Recent work has shown promise by incorporating pixel-level visual information into the reasoning process, enabling VLMs to access high-resolution visual details during their thought process. However, this pixel-level information is often overused, leading to inefficiency and distraction from irrelevant visual details. To address these challenges, we propose the first framework for adaptive pixel reasoning that dynamically determines necessary pixel-level operations based on the input query. Specifically, we first apply operation-aware supervised fine-tuning to establish baseline competence in textual reasoning and visual operations, then design a novel rollout-guided reinforcement learning framework relying on feedback of the model's own responses, which enables the VLM to determine when pixel operations should be invoked based on query difficulty. Experiments on extensive multimodal reasoning benchmarks show that our model achieves superior performance while significantly reducing unnecessary visual operations. Impressively, our model achieves 73.4\% accuracy on HR-Bench 4K while maintaining a tool usage ratio of only 20.1\%, improving accuracy and simultaneously reducing tool usage by 66.5\% compared to the previous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。