提出分级语境相似度匹配,让图文模型理解描述的层次性。
Descriptive Image-Text Matching with Graded Contextual Similarity
- 用累积TF-IDF衡量句子描述度,动态调整正负样本连接
- 在三个数据集上超越现有方法,发现更多潜在匹配对
- 适合需要理解描述层次与灵活对应关系的研究者
图像-文本匹配旨在通过学习视觉与文本间的成对相似性建立对应关系。现有方法多采用稀疏二值监督,仅判断图文是否匹配,忽略了描述中固有的多对多关系——一张图可用不同描述层级的多种句子表达。同时,现有方法忽视了从一般到具体的隐含关联,而这正是多对多关系的底层逻辑。本文提出描述性图像-文本匹配(DITM),通过探索语言的描述灵活性,学习图像与文本间的分级上下文相似性。利用累积词频-逆文档频率(TF-IDF)构建句子描述度评分,以平衡关键词对相似性的贡献。DITM通过两个关键机制提升匹配能力:(1) 动态放松正负样本间连接,修正误判负样本;(2) 按从泛化到具体顺序对相关句子进行对齐,实现更精确匹配。相比严格的二值监督,DITM增强了对最优匹配和潜在正例的发现能力。在MS-COCO、Flickr30K和CxC数据集上的大量实验表明,该方法在表征复杂图文关系方面优于当前最优模型。此外,基于HierarCaps基准的分析验证了DITM在层次推理能力上的显著提升。
原文摘要 · Abstract (English)
Image-text matching aims to build correspondences between visual and textual data by learning their pairwise similarities. Most existing approaches have adopted sparse binary supervision, indicating whether a pair of images and sentences matches or not. However, such sparse supervision covers a limited subset of image-text relationships, neglecting their inherent many-to-many correspondences; an image can be described in numerous texts at different descriptive levels. Moreover, existing approaches overlook the implicit connections from general to specific descriptions, which form the underlying rationale for the many-to-many relationships between vision and language. In this work, we propose descriptive image-text matching, called DITM, to learn the graded contextual similarity between image and text by exploring the descriptive flexibility of language. We formulate the descriptiveness score of each sentence with cumulative term frequency-inverse document frequency (TF-IDF) to balance the pairwise similarity according to the keywords in the sentence. Our method leverages sentence descriptiveness to learn robust image-text matching in two key ways: (1) to refine the false negative labeling, dynamically relaxing the connectivity between positive and negative pairs, and (2) to build more precise matching, aligning a set of relevant sentences in a generic-to-specific order. By moving beyond rigid binary supervision, DITM enhances the discovery of both optimal matches and potential positive pairs. Extensive experiments on MS-COCO, Flickr30K, and CxC datasets demonstrate the effectiveness of our method in representing complex image-text relationships compared to state-of-the-art approaches. In addition, DITM enhances the hierarchical reasoning ability of the model, supported by the extensive analysis on HierarCaps benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。