通过多尺度融合与检测引导，提升大模型对高分辨率图像的理解能力
MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
- 采用多分辨率语义融合，减少单尺度理解偏差和物体碎片化
- 引入开放词汇检测作为定位先验，显著提升目标召回率与准确性
- 无需训练即可兼容多种大模型，适合高分辨图像分析场景
高分辨率图像理解仍是多模态大语言模型（MLLMs）的关键挑战。现有方法依赖基于视觉的检索增强生成（RAG），从高分辨率图像中检索与查询相关的局部区域，以增强模型理解能力。然而，该范式常导致物体碎片化，引发语义偏差和检索不完整，且易受无关背景块干扰产生误检。为此，我们提出无需训练的多分辨率检索-检测融合框架（MRD），从局部与全局两个层面提升高分辨率图像理解能力。局部上，通过多分辨率语义融合实现跨尺度语义一致性，缓解单分辨率偏差并减轻物体碎片化问题；全局上，在统一框架内集成开放词汇目标检测（OVD）作为定位先验。在多个高分辨率图像基准测试上，针对多种MLLM的大量实验表明，MRD在单目标与多目标理解任务中均达到当前最优性能。代码将公开于：https://github.com/yf0412/MRD。
原文摘要 · Abstract (English)
Understanding high-resolution (HR) images remains a critical challenge for multimodal large language models (MLLMs). Recent approaches leverage vision-based retrieval-augmented generation (RAG) to retrieve query-relevant crops from HR images, improving understanding capacity of MLLMs. However, this paradigm often leads to object fragmentation, resulting in semantic bias and incomplete retrieval, while also introducing false positives from irrelevant background patches. To address these issues, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework that enhances HR image understanding from both local and global perspectives. Locally, MRD enforces cross-scale semantic consistency via multi-resolution semantic fusion to mitigate single-resolution bias and alleviate object fragmentation. Globally, it integrates open-vocabulary object detection (OVD) as localization priors within a unified framework. Extensive experiments across multiple MLLMs on HR image benchmarks demonstrate that MRD achieves state-of-the-art (SOTA) performance on both single-object and multi-object understanding tasks. Code will be available at: https://github.com/yf0412/MRD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。