无需训练，通过局部放大与自我修正提升模型对高分辨率图像的理解能力
Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement
- 先定位关键区域，再提取高清细节进行推理
- 在两个高分辨率多模态基准上显著提升准确率
- 适合需要精细视觉理解的场景，如医学影像分析
多模态大语言模型（MLLM）在解析高分辨率图像时常因细节缺失而表现不佳，尤其在复杂视觉理解任务中。本文提出无需训练的Zoom-Refine方法，通过协同的‘局部放大’与‘自我修正’机制增强模型能力。在‘局部放大’阶段，模型基于初始问答预测最相关的图像区域边界框；在‘自我修正’阶段，将该区域的高分辨率图像细粒度信息与原推理过程融合，重新评估并优化初步答案。该方法充分利用了模型本身的空间定位、上下文推理和对比分析能力，无需额外训练或外部专家。在两个高分辨率多模态基准上的实验表明，Zoom-Refine显著提升了模型性能。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLM) often struggle to interpret high-resolution images accurately, where fine-grained details are crucial for complex visual understanding. We introduce Zoom-Refine, a novel training-free method that enhances MLLM capabilities to address this issue. Zoom-Refine operates through a synergistic process of \textit{Localized Zoom} and \textit{Self-Refinement}. In the \textit{Localized Zoom} step, Zoom-Refine leverages the MLLM to provide a preliminary response to an input query and identifies the most task-relevant image region by predicting its bounding box coordinates. During the \textit{Self-Refinement} step, Zoom-Refine then integrates fine-grained details from the high-resolution crop (identified by \textit{Localized Zoom}) with its initial reasoning to re-evaluate and refine its preliminary response. Our method harnesses the MLLM's inherent capabilities for spatial localization, contextual reasoning and comparative analysis without requiring additional training or external experts. Comprehensive experiments demonstrate the efficacy of Zoom-Refine on two challenging high-resolution multimodal benchmarks. Code is available at \href{https://github.com/xavier-yu114/Zoom-Refine}{\color{magenta}github.com/xavier-yu114/Zoom-Refine}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。