用4.7%参数量实现文本检索行人最优效果,高效迁移多模态知识。
UP-Person: Unified Parameter-Efficient Transfer Learning for Text-based Person Retrieval
- 融合前缀、低秩和适配器三类轻量模块,统一提升特征提取能力。
- 在CUHK-PEDES等数据集上达到当前最佳性能,仅微调4.7%参数。
- 适合资源受限场景下高效部署文本检索行人系统,兼顾精度与效率。
文本驱动的行人检索(TPR)作为多模态任务,旨在根据文本描述从候选图像中检索目标行人。近年来,得益于对比视觉-语言预训练模型的发展,该任务受到广泛关注。以往方法利用CLIP提取视觉与文本特征,并对整个网络进行全量微调,虽显著优于单模态预训练模型,但易过拟合且损害泛化能力。本文提出一种统一的参数高效迁移学习方法(UP-Person),以充分迁移CLIP中的多模态知识。具体地,UP-Person同时集成三种轻量级参数高效微调组件:前缀(Prefix)、LoRA与适配器(Adapter)。其中,前缀与LoRA协同挖掘局部任务相关特征,适配器则用于调整全局特征分布。此外,两个基础模块经优化以适配统一架构:S-Prefix增强前缀注意力并改善梯度传播,提升前缀灵活性;L-Adapter与层归一化并行设计,缓解多子模块间重叠与交互冲突。大量实验表明,所提方法在CUHK-PEDES、ICFG-PEDES和RSTPReid等多个数据集上均达最优性能,且仅需微调4.7%参数。代码已开源。
原文摘要 · Abstract (English)
Text-based Person Retrieval (TPR) as a multi-modal task, which aims to retrieve the target person from a pool of candidate images given a text description, has recently garnered considerable attention due to the progress of contrastive visual-language pre-trained model. Prior works leverage pre-trained CLIP to extract person visual and textual features and fully fine-tune the entire network, which have shown notable performance improvements compared to uni-modal pre-training models. However, full-tuning a large model is prone to overfitting and hinders the generalization ability. In this paper, we propose a novel Unified Parameter-Efficient Transfer Learning (PETL) method for Text-based Person Retrieval (UP-Person) to thoroughly transfer the multi-modal knowledge from CLIP. Specifically, UP-Person simultaneously integrates three lightweight PETL components including Prefix, LoRA and Adapter, where Prefix and LoRA are devised together to mine local information with task-specific information prompts, and Adapter is designed to adjust global feature representations. Additionally, two vanilla submodules are optimized to adapt to the unified architecture of TPR. For one thing, S-Prefix is proposed to boost attention of prefix and enhance the gradient propagation of prefix tokens, which improves the flexibility and performance of the vanilla prefix. For another thing, L-Adapter is designed in parallel with layer normalization to adjust the overall distribution, which can resolve conflicts caused by overlap and interaction among multiple submodules. Extensive experimental results demonstrate that our UP-Person achieves state-of-the-art results across various person retrieval datasets, including CUHK-PEDES, ICFG-PEDES and RSTPReid while merely fine-tuning 4.7\% parameters. Code is available at https://github.com/Liu-Yating/UP-Person.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。