用生成图像增强文本与图像对齐,提升跨模态检索效果
GEA: Generation-Enhanced Alignment for Text-to-Image Person Retrieval
- 用扩散模型生成中间图像,丰富文本语义表示
- 在三个数据集上均超越现有方法,最高提升4.8% mAP
- 适合关注跨模态对齐与生成式融合的研究者
文本到图像的人体检索(TIPR)旨在根据自然语言描述检索人物图像。尽管许多TIPR方法已取得良好效果,但文本查询常无法准确全面反映图像内容,导致跨模态对齐不佳并过度拟合有限数据集。此外,文本与图像之间的固有模态差距进一步加剧了这些问题,使精准的跨模态检索更具挑战性。为此,我们从生成视角提出生成增强对齐(GEA)。GEA包含两个并行模块:(1) 文本引导标记增强(TGTE),引入扩散生成的图像作为中间语义表示,以弥合文本与视觉模式间的鸿沟。这些生成图像丰富了文本的语义表示,促进跨模态对齐。(2) 生成中间融合(GIF),通过生成图像、原始图像与文本特征间的交叉注意力融合,生成统一表征,并以三元组对齐损失优化。我们在三个公开TIPR数据集——CUHK-PEDES、RSTPReid和ICFG-PEDES上进行大量实验,验证了该方法的有效性。更多实现细节与扩展结果见https://github.com/sugelamyd123/Sup-for-GEA。
原文摘要 · Abstract (English)
Text-to-Image Person Retrieval (TIPR) aims to retrieve person images based on natural language descriptions. Although many TIPR methods have achieved promising results, sometimes textual queries cannot accurately and comprehensively reflect the content of the image, leading to poor cross-modal alignment and overfitting to limited datasets. Moreover, the inherent modality gap between text and image further amplifies these issues, making accurate cross-modal retrieval even more challenging. To address these limitations, we propose the Generation-Enhanced Alignment (GEA) from a generative perspective. GEA contains two parallel modules: (1) Text-Guided Token Enhancement (TGTE), which introduces diffusion-generated images as intermediate semantic representations to bridge the gap between text and visual patterns. These generated images enrich the semantic representation of text and facilitate cross-modal alignment. (2) Generative Intermediate Fusion (GIF), which combines cross-attention between generated images, original images, and text features to generate a unified representation optimized by triplet alignment loss. We conduct extensive experiments on three public TIPR datasets, CUHK-PEDES, RSTPReid, and ICFG-PEDES, to evaluate the performance of GEA. The results justify the effectiveness of our method. More implementation details and extended results are available at https://github.com/sugelamyd123/Sup-for-GEA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。