提出Signal框架,解决多模态目标重识别中的干扰与对齐问题。
Signal: Selective Interaction and Global-local Alignment for Multi-Modal Object Re-Identification
- 通过选择性交互模块筛选关键图像块,增强特征判别力。
- 在三个数据集上达到新高,最高提升1.8%的mAP。
- 适合关注多模态对齐与抗干扰的视觉检索研究者。
多模态目标重识别致力于利用互补的多模态图像信息检索特定对象。现有方法主要关注多模态特征融合,却忽视了背景干扰。此外,当前融合方法通常仅对模态对进行对齐,难以实现多模态一致性对齐。为此,我们提出一种名为Signal的新框架,包含选择性交互模块(SIM)和全局-局部对齐模块。SIM通过引入模态内与跨模态信息,筛选重要图像块并与其类别令牌交互,生成更具判别性的特征。全局对齐模块(GAM)通过最小化格拉姆空间中三维多面体体积,同步对齐多模态特征;局部对齐模块(LAM)以位移感知方式对齐局部特征。在三个多模态目标重识别基准(RGBNT201、RGBNT100、MSVR310)上的大量实验验证了该方法的有效性。代码已开源。
原文摘要 · Abstract (English)
Multi-modal object Re-IDentification (ReID) is devoted to retrieving specific objects through the exploitation of complementary multi-modal image information. Existing methods mainly concentrate on the fusion of multi-modal features, yet neglecting the background interference. Besides, current multi-modal fusion methods often focus on aligning modality pairs but suffer from multi-modal consistency alignment. To address these issues, we propose a novel selective interaction and global-local alignment framework called Signal for multi-modal object ReID. Specifically, we first propose a Selective Interaction Module (SIM) to select important patch tokens with intra-modal and inter-modal information. These important patch tokens engage in the interaction with class tokens, thereby yielding more discriminative features. Then, we propose a Global Alignment Module (GAM) to simultaneously align multi-modal features by minimizing the volume of 3D polyhedra in the gramian space. Meanwhile, we propose a Local Alignment Module (LAM) to align local features in a shift-aware manner. With these modules, our proposed framework could extract more discriminative features for object ReID. Extensive experiments on three multi-modal object ReID benchmarks (i.e., RGBNT201, RGBNT100, MSVR310) validate the effectiveness of our method. The source code is available at https://github.com/010129/Signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。