arXiv:2412.20646cs.CV2024-12被引 23

通过增强视觉细节理解,提升文本检索行人模型的准确率。

Enhancing Visual Representation for Text-based Person Searching

  • 用CLIP预训练模型结合文本引导的图像掩码任务,提升局部视觉特征提取能力。
  • 引入身份监督的全局特征校准,减少身份混淆,使检索准确率提升1%~9%。
  • 无需额外标注,适合追求高精度跨模态检索的研究者和开发者。

文本驱动的行人搜索旨在根据文本描述从大规模图像数据库中检索匹配的行人。该任务的核心挑战在于如何从行人图像和文本中提取有效细节,并在统一的潜在空间中实现跨模态对齐。以往方法采用在单模态数据上预训练的图像和文本编码器,分别提取全局与局部特征,并显式实现全局-局部对齐。然而,这些方法仍缺乏对视觉细节的理解能力,检索准确率受限于身份混淆问题。为此,本文重新思考视觉特征的重要性,提出VFE-TPS模型:利用预训练的多模态骨干网络CLIP学习基础多模态特征,并设计文本引导的掩码图像建模任务,以无标注方式增强模型对局部视觉细节的学习能力;同时引入身份监督的全局视觉特征校准任务,指导模型学习具有身份感知能力的全局特征。实验结果表明,在三个基准数据集上,所提模型显著优于现有方法,排名1(Rank-1)准确率提升约1%~9%。代码已开源。

原文摘要 · Abstract (English)

Text-based person search aims to retrieve the matched pedestrians from a large-scale image database according to the text description. The core difficulty of this task is how to extract effective details from pedestrian images and texts, and achieve cross-modal alignment in a common latent space. Prior works adopt image and text encoders pre-trained on unimodal data to extract global and local features from image and text respectively, and then global-local alignment is achieved explicitly. However, these approaches still lack the ability of understanding visual details, and the retrieval accuracy is still limited by identity confusion. In order to alleviate the above problems, we rethink the importance of visual features for text-based person search, and propose VFE-TPS, a Visual Feature Enhanced Text-based Person Search model. It introduces a pre-trained multimodal backbone CLIP to learn basic multimodal features and constructs Text Guided Masked Image Modeling task to enhance the model's ability of learning local visual details without explicit annotation. In addition, we design Identity Supervised Global Visual Feature Calibration task to guide the model learn identity-aware global visual features. The key finding of our study is that, with the help of our proposed auxiliary tasks, the knowledge embedded in the pre-trained CLIP model can be successfully adapted to text-based person search task, and the model's visual understanding ability is significantly enhanced. Experimental results on three benchmarks demonstrate that our proposed model exceeds the existing approaches, and the Rank-1 accuracy is significantly improved with a notable margin of about $1\%\sim9\%$. Our code can be found at https://github.com/zhangweifeng1218/VFE_TPS.

跨模态检索视觉理解行人搜索CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。