一个模型搞定多种视觉追踪任务，效率高且抗缺失模态。
Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
- 用双专家混合架构融合多模态信息，统一建模不同输入方式。
- 在12个基准上达到顶尖性能，压缩后仍保持强效果。
- 适合需要跨模态兼容与高效推理的追踪应用。
多模态视觉目标追踪根据输入模态可分为多种任务（如RGB和RGB+X追踪）。现有方法通常为每种模态训练独立模型或依赖预训练模型适配新模态，限制了效率、可扩展性和实用性。为此，我们提出OneTrackerV2，一种统一的多模态追踪框架，支持任意模态的端到端训练。我们设计了Meta Merger，将多模态信息嵌入统一空间，实现灵活融合与鲁棒性。进一步提出双混合专家（DMoE）：T-MoE建模时空关系用于追踪，M-MoE嵌入多模态知识，解耦跨模态依赖并减少特征冲突。采用共享架构、统一参数和单次端到端训练，OneTrackerV2在五个RGB及RGB+X追踪任务、12个基准上均达当前最优表现，同时保持高推理效率。值得注意的是，即使经过模型压缩，其性能依然强劲。此外，OneTrackerV2在模态缺失场景下表现出卓越鲁棒性。
原文摘要 · Abstract (English)
Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。