让多模态大模型智能决定何时何地看图，提升细粒度理解效率。
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
- 通过信心判断是否需要深入看图，避免无效计算。
- 在高分辨率图像上实现更准的视觉定位，准确率显著提升。
- 无需训练，适配多种模型，推理速度提升4倍以上。
多模态大语言模型正转向'以图思考'，主动探索图像细节。尽管有效，大规模训练成本高昂，促使人们关注轻量级、免训练方案。然而现有免训练方法存在两大缺陷：随意裁剪导致感知冗余，增加计算负担并引入噪声；语义意图与空间注意力偏离，难以精准定位用户关注区域。为此，我们提出LookWise框架，实现自适应视觉推理。该框架采用两阶段流程：基于置信度的模块决定何时需仔细观察，语义引导定位模块确定何处应聚焦。此设计使MLLM能在不额外训练的前提下，自适应获取细粒度视觉证据。在细粒度与高分辨率视觉推理基准测试中，LookWise持续优于强基线，且相比搜索式方法ZoomEye实现约4.0倍推理加速，展现出强大的跨模型泛化能力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details. While effective, large-scale training is computationally expensive, which has spurred growing interest in lightweight, training-free solutions. However, existing training-free methods suffer from two flaws: perceptual redundancy from indiscriminate cropping, which increases computational cost and introduces noise; and a drift between semantic intent and spatial attention, which prevents accurate localization of user-focused regions. To address these challenges, we propose LookWise, a framework for adaptive visual reasoning. LookWise follows a two-stage pipeline: a confidence-based module decides when to look more carefully, and a semantic-guided localization module determines where to look. This design enables MLLMs to adaptively acquire fine-grained visual evidence without additional training. Experiments on fine-grained and high-resolution visual reasoning benchmarks show that LookWise consistently improves accuracy over strong baselines while achieving an approximately $4.0\times$ inference speedup over the search-based method ZoomEye, demonstrating robust cross-model generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。