用Mamba提升多模态物体重识别的效率与精度
MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt
- 引入Mamba聚合模块处理跨模态长序列数据
- 在三个数据集上实现最优性能，参数量更低
- 适合做多模态视觉任务的工程师和研究者
多模态物体重识别（ReID）旨在利用不同模态的图像信息检索特定物体。尽管大规模预训练模型如CLIP在单模态物体重识别中表现优异，但在多模态场景下尚未被充分探索。现有方法在处理跨模态长序列时存在明显局限。为此，本文提出MambaPro框架：首先使用并行前馈适配器（PFA）将CLIP适配至多模态任务；其次设计协同残差提示（SRP）引导多模态特征联合学习；最后利用Mamba在长序列上的卓越扩展性，引入Mamba聚合（MA）高效建模模态间交互。实验在RGBNT201、RGBNT100和MSVR310三个基准上验证了方法有效性，显著提升了特征鲁棒性且计算复杂度更低。
原文摘要 · Abstract (English)
Multi-modal object Re-IDentification (ReID) aims to retrieve specific objects by utilizing complementary image information from different modalities. Recently, large-scale pre-trained models like CLIP have demonstrated impressive performance in traditional single-modal object ReID tasks. However, they remain unexplored for multi-modal object ReID. Furthermore, current multi-modal aggregation methods have obvious limitations in dealing with long sequences from different modalities. To address above issues, we introduce a novel framework called MambaPro for multi-modal object ReID. To be specific, we first employ a Parallel Feed-Forward Adapter (PFA) for adapting CLIP to multi-modal object ReID. Then, we propose the Synergistic Residual Prompt (SRP) to guide the joint learning of multi-modal features. Finally, leveraging Mamba's superior scalability for long sequences, we introduce Mamba Aggregation (MA) to efficiently model interactions between different modalities. As a result, MambaPro could extract more robust features with lower complexity. Extensive experiments on three multi-modal object ReID benchmarks (i.e., RGBNT201, RGBNT100 and MSVR310) validate the effectiveness of our proposed methods. The source code is available at https://github.com/924973292/MambaPro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。