让多模态大模型像人一样局部放大看图，提升细粒度视觉推理能力。
ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration
- 将图像构建成树状结构，逐级局部放大搜索关键区域。
- 在高分辨率基准上，模型性能提升超15%，小模型超越GPT-4o。
- 无需训练、兼容多种模型，适合复杂视觉任务场景。
多模态大语言模型（MLLM）在视觉语言理解方面表现出色。近期，结合测试时扩展技术，这些模型在视觉推理上也展现出巨大潜力。然而，现有方法仍以文本层面为主：模型通过语言模型尝试不同文本组合，而视觉输入始终固定。这一范式限制了模型对丰富视觉信息的利用，尤其在包含大量细粒度元素的图像中，视觉层面的动态推理至关重要——即模型需动态聚焦图像特定区域，获取细节线索以准确决策。本文提出 Zoom Eye，一种无需训练、与模型无关的树搜索算法，专为视觉层面推理设计。该方法将图像视为分层树结构，每个子节点代表父节点的局部放大区域，根节点对应整张图像。算法使 MLLM 能模拟人类缩放行为，从根节点到叶节点导航，寻找任务相关视觉证据。我们在一系列高分辨率基准上进行实验，结果表明，Zoom Eye 显著提升了多个 MLLM 的性能（如 InternVL2.5-8B 在 HR-Bench 上分别提升 15.71% 和 17.69%），并使 3-8B 的小型模型超越 GPT-4o 等强大模型。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in vision-language understanding. Recently, with the integration of test-time scaling techniques, these models have also shown strong potential in visual reasoning. However, most existing reasoning approaches remain text-level in nature: MLLMs are prompted to explore various combinations of textual tokens via their underlying language model, while the visual input remains fixed throughout the reasoning process. This paradigm limits the model's ability to fully exploit rich visual information, particularly when dealing with images containing numerous fine-grained elements. In such cases, vision-level reasoning becomes crucial - where models dynamically zoom into specific regions of the image to gather detailed visual cues necessary for accurate decision-making. In this paper, we propose Zoom Eye, a training-free, model-agnostic tree search algorithm tailored for vision-level reasoning. Zoom Eye treats an image as a hierarchical tree structure, where each child node represents a zoomed-in sub-region of its parent, and the root corresponds to the full image. The algorithm enables MLLMs to simulate human-like zooming behavior by navigating from root to leaf nodes in search of task-relevant visual evidence. We experiment on a series of high-resolution benchmarks and the results demonstrate that Zoom Eye consistently improves the performance of multiple MLLMs by a large margin (e.g., InternVL2.5-8B increases by 15.71% and 17.69% on HR-Bench) and also enables small 3-8B MLLMs to outperform strong large models such as GPT-4o. Code: https://github.com/om-ai-lab/ZoomEye
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。