不训练不微调,让多模态大模型在高分辨率图像上表现更好。
A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Images
- 先用低分辨率图找候选区域,再基于候选区精确预测
- 4K GUI定位任务提升21.3%,8K图像感知任务提升5.2%
- 无需训练,适配各类任务,适合部署高分辨率应用
多模态大语言模型(MLLM)在视觉-语言理解、推理和生成方面表现出色,但在需要精细定位和推理的高分辨率图像任务中表现不佳。根源在于其微调时使用固定图像分辨率,与预训练图像编码器对齐,导致输入高分辨率图像时因训练-测试分辨率不一致而泛化能力差;直接下采样虽能保持一致性,却损失细节,降低性能。为此,我们提出无需训练、任务无关的两阶段框架ECP:首先利用低分辨率预测隐含的定位线索提取候选区域,再基于候选区域进行最终预测,有效保留细粒度视觉信息。在4K GUI定位和4K、8K MLLM感知任务上,相比基线分别提升21.3%、5.8%、5.2%绝对性能,验证了有效性。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in vision-language understanding, reasoning, and generation. However, they struggle with tasks requiring fine-grained localization and reasoning in high-resolution images. This constraint stems from the fact that MLLMs are fine-tuned with fixed image resolution to align with the pre-trained image encoder used in MLLM. Consequently, feeding high-resolution images directly into MLLMs leads to poor generalization due to a train-test resolution discrepancy, while downsampling these images-although ensuring consistency-compromises fine-grained visual details and ultimately degrades performance. To address this challenge, we propose Extract Candidate then Predict (ECP), a novel training-free, task-agnostic two-stage framework designed to enhance MLLM performance on high-resolution images. The key intuition behind ECP is that while MLLMs struggle with high-resolution images, their predictions on downsampled images still contain implicit localization cues. By first identifying candidate region using the coarse prediction and then predicting the final output based on candidate region, ECP effectively preserves fine-grained details while mitigating the challenges posed by high-resolution data. We validate our framework on 4K GUI grounding and 4K, 8K MLLM perception, achieving +21.3%, +5.8%, +5.2% absolute improvement compared to baseline respectively, demonstrating its effectiveness. Code is available at https://github.com/yenncye/ECP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。