提出混合专家适配器,高效融合多模态特征提升跟踪精度。
Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking
- 设计稀疏-稠密混合专家模块,分离建模模态特异性与共享信息。
- 在多个数据集上超越现有方法,最高提升1.8%成功率。
- 适合资源受限场景下的多模态跟踪应用。
参数高效微调(PEFT)技术如提示词和适配器被广泛应用于多模态跟踪,以缓解全模型微调带来的耗时、高资源消耗、参数存储压力及灾难性遗忘等问题。然而,由于跨模态异质性,现有基于PEFT的方法难以在统一框架下有效表征多模态特征。为此,本文提出一种新型的稀疏-稠密混合专家适配器(SDMoEA)框架,用于统一结构下的多模态跟踪。具体地,设计了包含稀疏专家模块与稠密共享专家模块的SDMoE,分别捕捉模态特异性信息与跨模态共享信息。此外,为克服现有方法在多层次多模态融合中建模高阶相关性的局限,引入基于格拉姆矩阵的语义对齐超图融合(GSAHF)模块。该模块首先利用格拉姆矩阵实现跨模态语义对齐,确保构建的超图准确反映模态间的语义相似性与高阶依赖关系;随后将对齐特征融入超图结构,发挥其建模高阶关系的能力,实现多层次多模态信息的深度融合。大量实验表明,所提方法在多个多模态跟踪基准上表现优异,包括LasHeR、RGBT234、VTUAV、VisEvent、COESOT、DepthTrack和VOT-RGBD2022,显著优于其他PEFT方法。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) techniques, such as prompts and adapters, are widely used in multi-modal tracking because they alleviate issues of full-model fine-tuning, including time inefficiency, high resource consumption, parameter storage burden, and catastrophic forgetting. However, due to cross-modal heterogeneity, most existing PEFT-based methods struggle to effectively represent multi-modal features within a unified framework with shared parameters. To address this problem, we propose a novel Sparse-Dense Mixture of Experts Adapter (SDMoEA) framework for PEFT-based multi-modal tracking under a unified model structure. Specifically, we design an SDMoE module as the multi-modal adapter to model modality-specific and shared information efficiently. SDMoE consists of a sparse MoE and a dense-shared MoE: the former captures modality-specific information, while the latter models shared cross-modal information. Furthermore, to overcome limitations of existing tracking methods in modeling high-order correlations during multi-level multi-modal fusion, we introduce a Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module. It first employs Gram matrices for cross-modal semantic alignment, ensuring that the constructed hypergraph accurately reflects semantic similarity and high-order dependencies between modalities. The aligned features are then integrated into the hypergraph structure to exploit its ability to model high-order relationships, enabling deep fusion of multi-level multi-modal information. Extensive experiments demonstrate that the proposed method achieves superior performance compared with other PEFT approaches on several multi-modal tracking benchmarks, including LasHeR, RGBT234, VTUAV, VisEvent, COESOT, DepthTrack, and VOT-RGBD2022.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。