提出轻量级提示调优框架,实现多模态目标重识别的高效训练。
DMPT: Decoupled Modality-aware Prompt Tuning for Multi-modal Object Re-identification
- 解耦视觉提示为模态特异性与语义共享提示,仅优化少量参数。
- 在多个基准上达到顶尖性能,仅需6.5%骨干网络参数微调。
- 适合资源受限场景,尤其适用于红外、可见光等多模态数据。
基于大规模预训练骨干网络(如ViT)的多模态目标重识别方法已取得显著进展。然而,这些方法通常采用全量微调,需优化大量骨干参数,带来高昂的计算与存储开销。本文提出一种专为多模态目标重识别设计的高效提示调优框架DMPT,冻结主干网络,仅优化新增的解耦模态感知参数。具体地,将视觉提示显式解耦为模态特定提示(利用强大文本编码器的先验知识)和模态无关语义提示(从可见光、近红外、热红外等多模态输入中提取语义信息)。在此基础上,设计提示逆绑定(PromptIBind)策略,以绑定提示为媒介连接不同模态的语义提示令牌,促进互补多模态信息交换,提升最终重识别效果。在多个常用基准上的实验表明,DMPT可达到与现有最先进方法相当的性能,同时仅需6.5%骨干网络参数微调。
原文摘要 · Abstract (English)
Current multi-modal object re-identification approaches based on large-scale pre-trained backbones (i.e., ViT) have displayed remarkable progress and achieved excellent performance. However, these methods usually adopt the standard full fine-tuning paradigm, which requires the optimization of considerable backbone parameters, causing extensive computational and storage requirements. In this work, we propose an efficient prompt-tuning framework tailored for multi-modal object re-identification, dubbed DMPT, which freezes the main backbone and only optimizes several newly added decoupled modality-aware parameters. Specifically, we explicitly decouple the visual prompts into modality-specific prompts which leverage prior modality knowledge from a powerful text encoder and modality-independent semantic prompts which extract semantic information from multi-modal inputs, such as visible, near-infrared, and thermal-infrared. Built upon the extracted features, we further design a Prompt Inverse Bind (PromptIBind) strategy that employs bind prompts as a medium to connect the semantic prompt tokens of different modalities and facilitates the exchange of complementary multi-modal information, boosting final re-identification results. Experimental results on multiple common benchmarks demonstrate that our DMPT can achieve competitive results to existing state-of-the-art methods while requiring only 6.5% fine-tuning of the backbone parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。