arXiv:2410.09382cs.CV2024-10被引 6

用合成描述提升行人重识别,让模型更懂图像里的细节。

CLIP-SCGI: Synthesized Caption-Guided Inversion for Person Re-Identification

  • 用大模型生成行人图像的伪描述,变单模为双模训练。
  • 在四个主流数据集上显著超越当前最佳方法。
  • 适合关注视觉语言对齐与行人识别的研究者。

行人重识别(ReID)近期受益于 CLIP 等大规模预训练视觉-语言模型。然而,缺乏具体描述导致依赖隐式文本嵌入,需复杂且低效的训练策略。为此,我们提出一种简单方案:利用现有图像描述模型生成行人图像的伪描述,从而借助大型视觉-语言模型提升 ReID 性能。通过 LLAVA 等模型,基于固定模板生成包含性别、衣着、年龄等关键语义属性的高质量描述。将训练集从单模态(图像)扩展为双模态(图像+文本),提出 CLIP-SCGI 框架,利用合成描述引导学习判别性与鲁棒表示。该框架基于 CLIP,通过两个模块融合图像与文本嵌入。为解决生成描述质量问题,引入描述引导反演模块,将图像中的相关视觉信息转换为伪词令牌,以捕捉语义属性。提取特征后,经跨模态融合模块引导模型关注与描述语义一致的区域,优化视觉编码器以提取更具判别性的特征。在四个主流 ReID 基准测试中,CLIP-SCGI 显著优于现有最优方法。

原文摘要 · Abstract (English)

Person re-identification (ReID) has recently benefited from large pretrained vision-language models such as Contrastive Language-Image Pre-Training (CLIP). However, the absence of concrete descriptions necessitates the use of implicit text embeddings, which demand complicated and inefficient training strategies. To address this issue, we first propose one straightforward solution by leveraging existing image captioning models to generate pseudo captions for person images, and thereby boost person re-identification with large vision language models. Using models like the Large Language and Vision Assistant (LLAVA), we generate high-quality captions based on fixed templates that capture key semantic attributes such as gender, clothing, and age. By augmenting ReID training sets from uni-modality (image) to bi-modality (image and text), we introduce CLIP-SCGI, a simple yet effective framework that leverages synthesized captions to guide the learning of discriminative and robust representations. Built on CLIP, CLIP-SCGI fuses image and text embeddings through two modules to enhance the training process. To address quality issues in generated captions, we introduce a caption-guided inversion module that captures semantic attributes from images by converting relevant visual information into pseudo-word tokens based on the descriptions. This approach helps the model better capture key information and focus on relevant regions. The extracted features are then utilized in a cross-modal fusion module, guiding the model to focus on regions semantically consistent with the caption, thereby facilitating the optimization of the visual encoder to extract discriminative and robust representations. Extensive experiments on four popular ReID benchmarks demonstrate that CLIP-SCGI outperforms the state-of-the-art by a significant margin.

行人重识别视觉语言模型合成描述跨模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。