通过粗到细的视觉理解框架，提升多模态大模型对细微区域的感知能力。
CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
- 分两阶段：先定位大致区域，再通过视觉提示增强关键区域注意力。
- 在多个基准上显著提升基线模型性能，尤其在细粒度任务中效果明显。
- 适合需要精准视觉理解的多模态应用，如图像问答与细节识别。
大型语言模型（LLM）的出色表现推动了多模态大语言模型（MLLM）的发展，后者在多种多模态任务中展现出巨大潜力。然而，现有MLLM在处理细粒度多模态挑战时仍面临困难。我们认为，这一局限性与模型的视觉定位能力密切相关。视觉编码器受限的空间感知能力和感知敏锐度，常导致图像中无关背景信息干扰，使模型忽略细微但关键的细节，从而难以实现精细的区域视觉理解。为此，本文将多模态理解分解为从粗到细（CoF）两个阶段：第一阶段，引导MLLM定位答案的大致区域；第二阶段，通过视觉提示工程强化模型对图像相关区域的关注，调整其注意力权重。这有效提升了视觉定位精度和下游任务的整体表现。实验表明，该方法显著提升了基线模型性能，具备良好的泛化能力与有效性。CoF方法开源地址：https://github.com/Gavin001201/CoF。
原文摘要 · Abstract (English)
The impressive performance of Large Language Model (LLM) has prompted researchers to develop Multi-modal LLM (MLLM), which has shown great potential for various multi-modal tasks. However, current MLLM often struggles to effectively address fine-grained multi-modal challenges. We argue that this limitation is closely linked to the models' visual grounding capabilities. The restricted spatial awareness and perceptual acuity of visual encoders frequently lead to interference from irrelevant background information in images, causing the models to overlook subtle but crucial details. As a result, achieving fine-grained regional visual comprehension becomes difficult. In this paper, we break down multi-modal understanding into two stages, from Coarse to Fine (CoF). In the first stage, we prompt the MLLM to locate the approximate area of the answer. In the second stage, we further enhance the model's focus on relevant areas within the image through visual prompt engineering, adjusting attention weights of pertinent regions. This, in turn, improves both visual grounding and overall performance in downstream tasks. Our experiments show that this approach significantly boosts the performance of baseline models, demonstrating notable generalization and effectiveness. Our CoF approach is available online at https://github.com/Gavin001201/CoF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。