提出高效自适应多模态追踪器,解决性能与效率的矛盾。
SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

- 通过动态对齐跨模态注意力图,缓解模态偏差问题。
- 在红外、深度、可见光追踪任务中超越当前最优方法。
- 适合需要高效多模态追踪的工业级应用开发人员。
多模态追踪中的参数高效微调(PEFT)正面临性能提升伴随参数量激增的问题,严重削弱了其高效性承诺。本文提出SEATrack——一种简单、高效且自适应的双流多模态追踪器,从两个互补角度破解这一难题。首先,聚焦匹配响应的跨模态对齐,发现现有双流方法因模态特异性偏差导致注意力图冲突,阻碍联合表征学习。为此,提出AMG-LoRA,将低秩适配(LoRA)与自适应互引导(AMG)结合,动态优化并对齐跨模态注意力。其次,摒弃传统局部融合方式,引入分层专家混合(HMoE),实现高效全局关系建模,在表达能力与计算效率间取得平衡。实验表明,SEATrack在RGB-T、RGB-D和RGB-E追踪任务中均显著优于现有方法,兼顾性能与效率。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. \href{https://github.com/AutoLab-SAI-SJTU/SEATrack}{\textcolor{cyan}{Code is available}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。