通过分层对齐提升大模型对复杂视觉关系的理解能力
Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
- 构建从简单到复杂的多粒度跨模态对齐层次结构
- 在多个视觉推理任务上显著超越现有基线模型
- 适合研究视觉语言模型推理能力的开发者和研究人员
现有大型视觉语言模型（LVLMs）在跨模态概念匹配方面表现优异，但在组合性概念和实体间高层关系推理上存在不足。本文提出一种新型框架PromViL，通过构建从简单到复杂的多粒度跨模态对齐层次结构，使模型能够逐步将文本描述与对应视觉区域对齐，并利用低层级上下文信息支持高层推理。为支持该学习过程，我们基于Visual Genome构建了一个包含丰富嵌套组合性视觉-语言对的新数据集。实验结果表明，PromViL在多种视觉定位与组合问答任务上均显著优于基线模型。代码已公开于：https://github.com/lqh52/PromViL。
原文摘要 · Abstract (English)
Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。