无需标注数据，大模型也能自发理解图文对应关系。
Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
- 用注意力图实现像素级图像分割，挖掘模型隐含的视觉定位能力。
- 在无监督条件下达成44.2%的掩码召回率，优于有大量标注训练的GLaMM。
- 采用扩散模型视觉编码器，提升泛化性与可扩展性，适合通用视觉理解任务。
当前大型多模态模型（LMMs）在视觉定位任务中面临挑战，即需将语言成分与视觉实体关联。不同于常规通过额外定位监督微调的方法，我们发现即使在无显式定位监督训练下，定位能力仍可自发涌现。为揭示此现象，我们提出“关注并分割”方法，利用标准LMM的注意力图实现像素级分割。此外，为增强定位能力，我们设计DIFFLMM，采用基于扩散的视觉编码器替代传统CLIP编码器，并在相同弱监督条件下训练。该方法摆脱了定位专用数据集的偏差与规模限制，更具泛化性和可扩展性。在定位特定与通用视觉问答基准上均取得竞争力表现，分别优于定位型与通用型LMM。值得注意的是，在无任何定位监督的情况下，于定位对话生成任务上达到44.2%的掩码召回率，超越广泛监督的GLaMM模型。
原文摘要 · Abstract (English)
Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities. Contrary to the common practice that fine-tunes LMMs with additional grounding supervision, we find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision. To reveal this emerging grounding, we introduce an "attend-and-segment" method which leverages attention maps from standard LMMs to perform pixel-level segmentation. Furthermore, to enhance the grounding ability, we propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision. Without being constrained by the biases and limited scale of grounding-specific supervision data, our approach is more generalizable and scalable. We achieve competitive performance on both grounding-specific and general visual question answering benchmarks, compared with grounding LMMs and generalist LMMs, respectively. Notably, we achieve a 44.2 grounding mask recall on grounded conversation generation without any grounding supervision, outperforming the extensively supervised model GLaMM. Project page: https://GroundLMM-ICCV.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。