arXiv:2508.01592cs.CVcs.AI2025-08中稿 · ICRA

用双适配器实现高效多模态跟踪,参数仅0.93万却达顶尖性能。

DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter

  • 设计双适配器结构,分别处理时空特征与跨模态融合。
  • 在五个基准上实现当前最佳效果,仅需0.93万可训练参数。
  • 适合做轻量化多模态视觉跟踪的开发者和研究者参考。

本文探索适配器微调,提出一种新颖的双适配器架构——DMTrack,用于时空多模态跟踪。核心在于两个简单而有效的模块:时空模态适配器(STMA)和渐进式模态互补适配器(PMCA)。STMA独立应用于各模态,通过自提示机制调整冻结主干提取的时空特征,缓解模态间差异,促进跨模态融合。PMCA则通过像素级浅层与深层适配器渐进式引导跨模态提示:浅层适配器共享两模态参数,打通模态分支间信息流;深层适配器利用像素级内模态注意力调节初步融合信息,并通过像素级跨模态注意力生成模态感知提示。该设计使DMTrack仅用0.93万可训练参数即在五个基准上达到先进水平。实验验证其有效性,代码与模型将开源于https://github.com/Nightwatch-Fox11/DMTrack。

原文摘要 · Abstract (English)

In this paper, we explore adapter tuning and introduce a novel dual-adapter architecture for spatio-temporal multimodal tracking, dubbed DMTrack. The key of our DMTrack lies in two simple yet effective modules, including a spatio-temporal modality adapter (STMA) and a progressive modality complementary adapter (PMCA) module. The former, applied to each modality alone, aims to adjust spatio-temporal features extracted from a frozen backbone by self-prompting, which to some extent can bridge the gap between different modalities and thus allows better cross-modality fusion. The latter seeks to facilitate cross-modality prompting progressively with two specially designed pixel-wise shallow and deep adapters. The shallow adapter employs shared parameters between the two modalities, aiming to bridge the information flow between the two modality branches, thereby laying the foundation for following modality fusion, while the deep adapter modulates the preliminarily fused information flow with pixel-wise inner-modal attention and further generates modality-aware prompts through pixel-wise inter-modal attention. With such designs, DMTrack achieves promising spatio-temporal multimodal tracking performance with merely 0.93M trainable parameters. Extensive experiments on five benchmarks demonstrate that DMTrack achieves state-of-the-art results. Our code and models will be available at https://github.com/Nightwatch-Fox11/DMTrack.

多模态跟踪适配器轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。