让大模型像侦探一样分层探索高分辨率图像,自动定位关键细节。
Towards High-Resolution Visual Perception via Hierarchical Entity Exploration

- 不需训练,通过动态分层探索图像区域,按需聚焦细节。
- 在两个高分辨率基准上优于现有方法,准确率提升5.2%~8.7%。
- 适合处理复杂场景的视觉理解任务,尤其对多模态大模型有帮助。
高分辨率图像感知仍是多模态大语言模型(MLLMs)的关键挑战,因整体处理常丢失细粒度信息。现有方法或需训练引导关注位置,或固定分区,难以泛化于复杂高分辨率场景。本文提出无需训练、与模型无关的分层实体探索(HEE)框架,将静态图像理解转化为查询驱动的动态探索。HEE首先用双重评分机制评估各区域是否足以回答问题;若不足,则在最有希望区域中应用目标检测提取细粒度实体,聚类成连贯子区域,并构建多层级语义结构以深入探索。当深层区域仍无法给出置信答案时,信心引导的回溯机制会重访其他路径,确保自适应感知。大量实验表明,HEE在两种复杂高分辨率基准(Visual Probe 和 HR-Bench)上显著优于无训练方法如 ZoomEye 与 RAP,跨不同 MLLMs(如 Qwen2.5-VL 与 LLaVA-OneVision)表现更优,且在 MME-RealWorld 基准上展现良好泛化能力。
原文摘要 · Abstract (English)
High-resolution (HR) image perception remains a key challenge in multimodal large language models (MLLMs), as fine-grained details are often lost when the image is processed as a whole. Existing methods either require training to teach models where to look or heuristically divide the image into fixed regions, both of which struggle to generalize in complex HR scenes. In this work, we propose Hierarchical Entity Exploration (HEE), a training-free and model-agnostic framework that transforms static image understanding into dynamic, query-guided entity exploration. HEE first evaluates each region using a dual scoring mechanism to determine whether it already contains sufficient evidence to answer the question. If not, it applies object detection within the most promising region to extract fine-grained entities, clusters them into coherent subregions, and organizes them into a multi-level semantic hierarchy for deeper exploration. When deeper regions still fail to yield confident answers, a confidence-guided backtracking mechanism revisits alternative paths to ensure adaptive perception. Extensive results show that HEE outperforms training-free methods like ZoomEye and RAP in both accuracy and efficiency on two complex HR benchmarks (Visual Probe and HR-Bench), across different MLLMs such as Qwen2.5-VL and LLaVA-OneVision. Moreover, HEE demonstrates generalization on the MME-RealWorld benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。