提出全模式细粒度对齐框架,提升文本到图像人物检索精度
Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrieval
- 通过显式细粒度对齐和自适应相似度匹配,增强跨模态特征对齐
- 在三个公开数据集上达到当前最优全局匹配性能
- 适合关注跨模态对齐与检索精度的研究者
文本到图像人物检索(TIPR)是一项跨模态匹配任务,旨在根据文本描述识别最相关的图像。其核心挑战在于统一嵌入空间中实现文本与视觉模态的鲁棒对应。现有方法依赖注意力机制进行隐式局部对齐,但缺乏验证所有局部特征是否正确对齐的能力。此外,现有方法过度强调硬负样本以强化正负对区分,常忽略错误匹配的正样本。为此,本文提出FMFA框架,通过显式细粒度对齐与隐式关系推理相结合的“全模式”机制,在不引入额外监督信号的前提下提升全局匹配效果。具体提出自适应相似度分布匹配(A-SDM)模块,自适应拉近未正确对齐的正样本对;并设计显式细粒度对齐(EFA)模块,通过稀疏化相似度矩阵与硬编码方式增强跨模态局部交互。在三个公开数据集上的实验表明,该方法在所有全局匹配方法中达到领先性能。代码已开源:https://github.com/yinhao1102/FMFA。
原文摘要 · Abstract (English)
Text-to-Image Person Retrieval (TIPR) is a cross-modal matching task designed to identify the person images that best correspond to a given textual description. The key difficulty in TIPR is to realize robust correspondence between the textual and visual modalities within a unified latent representation space. To address this challenge, prior approaches incorporate attention mechanisms for implicit cross-modal local alignment. However, they lack the ability to verify whether all local features are correctly aligned. Moreover, existing methods tend to emphasize the utilization of hard negative samples during model optimization to strengthen discrimination between positive and negative pairs, often neglecting incorrectly matched positive pairs. To mitigate these problems, we propose FMFA, a cross-modal Full-Mode Fine-grained Alignment framework, which enhances global matching through explicit fine-grained alignment and existing implicit relational reasoning -- hence the term ``full-mode'' -- without introducing extra supervisory signals. In particular, we propose an Adaptive Similarity Distribution Matching (A-SDM) module to rectify unmatched positive sample pairs. A-SDM adaptively pulls the unmatched positive pairs closer in the joint embedding space, thereby achieving more precise global alignment. Additionally, we introduce an Explicit Fine-grained Alignment (EFA) module, which makes up for the lack of verification capability of implicit relational reasoning. EFA strengthens explicit cross-modal fine-grained interactions by sparsifying the similarity matrix and employs a hard coding method for local alignment. We evaluate our method on three public datasets, where it attains state-of-the-art results among all global matching methods. The code for our method is publicly accessible at https://github.com/yinhao1102/FMFA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。