arXiv:2603.21100cs.CVcs.AI2026-03

提出渐进式适配框架,提升红外/深度/事件等多模态跟踪性能。

Learning Progressive Adaptation for Multi-Modal Tracking

  • 分阶段引入模态专属、跨模态交互与任务适配器。
  • 在三类多模态追踪任务上均超越现有最优方法。
  • 适合需要融合异构传感器数据的视觉追踪场景。

由于成对多模态数据稀缺,现有方法通常基于预训练的RGB模型并采用参数高效微调模块构建多模态追踪器。然而,这些方法忽视了对预训练模型的深层适配,未能有效调节单一模态、跨模态交互及预测头。为此,本文提出渐进式多模态追踪适配框架(PATrack)。该框架集成模态依赖、模态纠缠与任务级适配器,通过渐进策略有效弥合预训练RGB网络与多模态数据间的差距。具体地,模态依赖适配器增强各模态的高低频特征,提升模态内表示鲁棒性;模态纠缠适配器引入由共享信息引导的交叉注意力机制,强化跨模态交互可靠性;同时针对预测头强归纳偏置不适应融合信息的问题,设计专用任务级适配器。实验在RGB+Thermal、RGB+Depth和RGB+Event三种追踪任务上验证,显著优于当前最优方法。代码已开源。

原文摘要 · Abstract (English)

Due to the limited availability of paired multi-modal data, multi-modal trackers are typically built by adopting pre-trained RGB models with parameter-efficient fine-tuning modules. However, these fine-tuning methods overlook advanced adaptations for applying RGB pre-trained models and fail to modulate a single specific modality, cross-modal interactions, and the prediction head. To address the issues, we propose to perform Progressive Adaptation for Multi-Modal Tracking (PATrack). This innovative approach incorporates modality-dependent, modality-entangled, and task-level adapters, effectively bridging the gap in adapting RGB pre-trained networks to multi-modal data through a progressive strategy. Specifically, modality-specific information is enhanced through the modality-dependent adapter, decomposing the high- and low-frequency components, which ensures a more robust feature representation within each modality. The inter-modal interactions are introduced in the modality-entangled adapter, which implements a cross-attention operation guided by inter-modal shared information, ensuring the reliability of features conveyed between modalities. Additionally, recognising that the strong inductive bias of the prediction head does not adapt to the fused information, a task-level adapter specific to the prediction head is introduced. In summary, our design integrates intra-modal, inter-modal, and task-level adapters into a unified framework. Extensive experiments on RGB+Thermal, RGB+Depth, and RGB+Event tracking tasks demonstrate that our method shows impressive performance against state-of-the-art methods. Code is available at https://github.com/ouha1998/Learning-Progressive-Adaptation-for-Multi-Modal-Tracking.

多模态目标追踪适配器跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。