让图像和文本更精准匹配特定实体,提升细粒度图文对齐能力。
EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning
- 用大模型生成解释文本作为桥梁,增强图文语义对齐。
- 在三个新闻数据集上显著超越现有方法,最高提升6.2%。
- 适合需要精准识别图像中具体对象的场景,如新闻分析。
近期图像-文本匹配研究进展显著,但主流模型多面向泛化查询,难以应对细粒度查询意图。本文提出面向特定实体的图像-文本匹配(EITM)任务,其核心挑战在于实体关联建模存在较大的语义鸿沟。为缩小这一鸿沟,我们以CLIP为基础,设计了一种多模态注意力对比学习框架,构建名为EntityCLIP的模型。关键创新在于利用预训练大语言模型(LLMs)生成解释性文本作为语义桥梁,将该解释文本与原始图像和文本一同输入自研的多模态注意力专家(MMAE)模块,有效融合解释信息,拉近实体相关图文在共享语义空间中的距离。在此基础上,进一步设计门控整合式图文匹配(GI-ITM)策略,通过自适应门控机制聚合特征,并施加图文匹配约束,引导精准对齐。在三个社交媒体新闻基准(N24News、VisualNews、GoodNews)上的大量实验表明,本方法显著优于现有方法,性能提升明显。
原文摘要 · Abstract (English)
Recent advancements in image-text matching have been notable, yet prevailing models predominantly cater to broad queries and struggle with accommodating fine-grained query intention. In this paper, we work towards the \textbf{E}ntity-centric \textbf{I}mage-\textbf{T}ext \textbf{M}atching (EITM), a task that the text and image involve specific entity-related information. The challenge of this task mainly lies in the larger semantic gap in entity association modeling, comparing with the general image-text matching problem.To narrow the huge semantic gap between the entity-centric text and the images, we take the fundamental CLIP as the backbone and devise a multimodal attentive contrastive learning framework to tam CLIP to adapt EITM problem, developing a model named EntityCLIP. The key of our multimodal attentive contrastive learning is to generate interpretive explanation text using Large Language Models (LLMs) as the bridge clues. In specific, we proceed by extracting explanatory text from off-the-shelf LLMs. This explanation text, coupled with the image and text, is then input into our specially crafted Multimodal Attentive Experts (MMAE) module, which effectively integrates explanation texts to narrow the gap of the entity-related text and image in a shared semantic space. Building on the enriched features derived from MMAE, we further design an effective Gated Integrative Image-text Matching (GI-ITM) strategy. The GI-ITM employs an adaptive gating mechanism to aggregate MMAE's features, subsequently applying image-text matching constraints to steer the alignment between the text and the image. Extensive experiments are conducted on three social media news benchmarks including N24News, VisualNews, and GoodNews, the results shows that our method surpasses the competition methods with a clear margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。