用文本桥接可见光与红外图像，提升跨模态行人重识别精度
CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification
- 通过生成可见光图像的文本语义，建立跨模态对齐基础
- 在红外图像特征中注入身份相关语义，提升共享编码器适应性
- 适合关注跨模态学习与红外行人识别的研究者
本文提出一种基于CLIP的新型可见-红外行人重识别方法CLIP4VI-ReID，包含文本语义生成（TSG）、红外特征嵌入（IFE）和高层语义对齐（HSA）三部分。针对自然图像与红外图像物理特性差异大问题，TSG仅对可见光图像生成文本语义，实现初步的可见-文本模态对齐；随后，IFE利用生成的文本语义校正红外图像特征嵌入，将身份相关语义注入共享图像编码器，增强其对红外模态的适应性，并实现间接的可见-红外模态对齐；最后，HSA进一步精炼高层语义对齐，确保微调后的文本语义仅含身份信息，从而实现更精确的跨模态对齐，提升所学共享表示的判别能力。大量实验证明，CLIP4VI-ReID在多个常用VI-ReID数据集上优于现有最先进方法。
原文摘要 · Abstract (English)
This paper proposes a novel CLIP-driven modality-shared representation learning network named CLIP4VI-ReID for VI-ReID task, which consists of Text Semantic Generation (TSG), Infrared Feature Embedding (IFE), and High-level Semantic Alignment (HSA). Specifically, considering the huge gap in the physical characteristics between natural images and infrared images, the TSG is designed to generate text semantics only for visible images, thereby enabling preliminary visible-text modality alignment. Then, the IFE is proposed to rectify the feature embeddings of infrared images using the generated text semantics. This process injects id-related semantics into the shared image encoder, enhancing its adaptability to the infrared modality. Besides, with text serving as a bridge, it enables indirect visible-infrared modality alignment. Finally, the HSA is established to refine the high-level semantic alignment. This process ensures that the fine-tuned text semantics only contain id-related information, thereby achieving more accurate cross-modal alignment and enhancing the discriminability of the learned modal-shared representations. Extensive experimental results demonstrate that the proposed CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on some widely used VI-ReID datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。