让模型从图像和标题中学会抽象概念,提升对高层次语义的理解能力。
Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision
- 设计分组对比损失,让同一组图文共享高层语义表示。
- 在MAGIC数据集上训练的CLEAR GLASS模型显著提升抽象概念识别能力。
- 无需直接标注概念,模型自发学习到高级语义,适合视觉理解研究者。
人类能识别图像为某一通用概念,而不仅限于物体及其关系。本文研究了视觉语言模型(VLM)在概念抽象方面的能力,并探索如何编码图像中的高层次概念信息,以增强模型的抽象能力。为此,我们构建了一个分组图文数据集MAGIC,其中每组包含若干图像、对应的标题及高层概念标签。提出一种新型分组对比损失:外层损失促使组内所有图文向共同的高层概念表示靠近;内层损失则约束组内图文间的距离。训练后,所提模型CLEAR GLASS展现出抽象概念识别能力,该能力是模型在未显式接触高层概念的情况下自发产生的。实验表明,该方法在抽象概念识别任务上优于当前最优模型。
原文摘要 · Abstract (English)
Humans can recognize an image as an instance of a general concept, beyond simply identifying its objects and their relationships. In this paper, we investigate 1. The extent to which VLMs have this concept abstraction capacity, and 2. Strategies for encoding the sort of higher-concept information in images that would enable the resulting VLM model (CLEAR GLASS model) to have this capability to a greater degree. To this end, we introduce a grouped image-caption dataset (MAGIC), which consists of several groups of image captions and for each group a set of associated images and higher-level conceptual labels. We use a novel contrastive loss technique to induce the model to encode in the representation of each image (caption) in a group the information that is common to all members of the image-caption group. Our main contribution is a grouped contrastive loss function based on text-image contrastive groups (outer contrastive loss) as well as an inner loss which measures the distances between image-caption instances in the group. Our training methodology results in the CLEAR GLASS model having the concept abstraction capacity as an emergent capacity because the model is not exposed to the higher-level concepts associated with each group. Instead, the training forces the model to create for each image-caption group a semantic representation that brings it closer to the semantic representation of the higher-level concepts in the latent semantic space. Our experiments show that this training methodology results in a model which shows improvement in abstract concept recognition compared to SOTA models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。