用语言提示统一可见光与红外行人特征,提升跨模态识别效果
Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification
- 基于CLIP生成跨模态共享语言提示,对齐不同视角行人特征
- 引入时空信息增强提示,显著提升在两个基准上的识别精度
- 适合关注跨模态行人重识别的视觉与多模态研究者
基于视频的可见光-红外行人重识别(VVI-ReID)旨在通过提取跨模态不变的序列级特征来匹配不同模态的行人序列。语言作为高层语义表征,可在可见光与红外模态中一致描述行人特征。利用对比语言图像预训练(CLIP)模型生成视频级语言提示并引导跨模态不变特征学习在理论上是可行的。然而,如何生成并利用跨模态共享的视频级语言提示以弥合模态差异仍是关键挑战。为此,我们提出一个简单而强大的框架——视频级语言驱动的VVI-ReID(VLD),包含两个核心模块:不变模态语言提示(IMLP)和时空提示(STP)。IMLP采用视觉编码器与提示学习器联合微调策略,有效生成跨模态共享文本提示,并在CLIP的多模态空间中将其与不同模态的视觉特征对齐,从而缓解模态差异。此外,STP通过时空枢纽(STH)和时空聚合(STA)两个子模块建模时空信息,进一步增强IMLP。STH在视觉变换器(ViT)各层中将时空信息聚合并扩散至每帧的[CLS] token;STA引入专用的身份级损失和多头注意力机制,确保STH聚焦于与身份相关的时空特征聚合。VLD框架在两个VVI-ReID基准上达到当前最优性能。代码将于https://github.com/Visuang/VLD发布。
原文摘要 · Abstract (English)
Video-based Visible-Infrared Person Re-Identification (VVI-ReID) aims to match pedestrian sequences across modalities by extracting modality-invariant sequence-level features. As a high-level semantic representation, language provides a consistent description of pedestrian characteristics in both infrared and visible modalities. Leveraging the Contrastive Language-Image Pre-training (CLIP) model to generate video-level language prompts and guide the learning of modality-invariant sequence-level features is theoretically feasible. However, the challenge of generating and utilizing modality-shared video-level language prompts to address modality gaps remains a critical problem. To address this problem, we propose a simple yet powerful framework, video-level language-driven VVI-ReID (VLD), which consists of two core modules: invariant-modality language prompting (IMLP) and spatial-temporal prompting (STP). IMLP employs a joint fine-tuning strategy for the visual encoder and the prompt learner to effectively generate modality-shared text prompts and align them with visual features from different modalities in CLIP's multimodal space, thereby mitigating modality differences. Additionally, STP models spatiotemporal information through two submodules, the spatial-temporal hub (STH) and spatial-temporal aggregation (STA), which further enhance IMLP by incorporating spatiotemporal information into text prompts. The STH aggregates and diffuses spatiotemporal information into the [CLS] token of each frame across the vision transformer (ViT) layers, whereas STA introduces dedicated identity-level loss and specialized multihead attention to ensure that the STH focuses on identity-relevant spatiotemporal feature aggregation. The VLD framework achieves state-of-the-art results on two VVI-ReID benchmarks. The code will be released at https://github.com/Visuang/VLD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。