提出自适应多模态跟踪框架,能动态应对传感器缺失数据。
What You Have is What You Track: Adaptive and Robust Multimodal Tracking
- 根据数据缺失率动态激活计算单元,灵活处理多模态输入。
- 在9个基准上达到最优,完整与缺失模态下均表现优异。
- 适合复杂场景下需鲁棒视觉跟踪的应用,如自动驾驶。
多模态数据可提升视觉跟踪对外观变化的鲁棒性,但传感器同步问题常导致数据缺失,尤其在视频场景中存在时间性短缺。尽管重要,该领域仍研究不足。本文首次系统研究了在时间不完整多模态数据下跟踪器的性能表现。结果显示,现有跟踪器因架构僵化,在缺失情况下性能显著下降。为此,我们提出一个灵活的鲁棒多模态跟踪框架。核心思想是:跟踪器应根据缺失数据率动态激活计算单元。通过新颖的异构专家混合融合机制(Heterogeneous Mixture-of-Experts)实现自适应复杂度,并结合视频级掩码策略,保障时序一致性与空间完整性,这对有效视频跟踪至关重要。令人惊讶的是,我们的模型不仅能适应不同缺失率,还能根据场景复杂度自动调整。大量实验表明,该模型在9个基准上达到当前最优(SOTA)表现,无论在常规完整数据或缺失模态设置下均表现出色。代码与基准数据将公开于 https://github.com/supertyd/FlexTrack/tree/main。
原文摘要 · Abstract (English)
Multimodal data is known to be helpful for visual tracking by improving robustness to appearance variations. However, sensor synchronization challenges often compromise data availability, particularly in video settings where shortages can be temporal. Despite its importance, this area remains underexplored. In this paper, we present the first comprehensive study on tracker performance with temporally incomplete multimodal data. Unsurprisingly, under such a circumstance, existing trackers exhibit significant performance degradation, as their rigid architectures lack the adaptability needed to effectively handle missing modalities. To address these limitations, we propose a flexible framework for robust multimodal tracking. We venture that a tracker should dynamically activate computational units based on missing data rates. This is achieved through a novel Heterogeneous Mixture-of-Experts fusion mechanism with adaptive complexity, coupled with a video-level masking strategy that ensures both temporal consistency and spatial completeness which is critical for effective video tracking. Surprisingly, our model not only adapts to varying missing rates but also adjusts to scene complexity. Extensive experiments show that our model achieves SOTA performance across 9 benchmarks, excelling in both conventional complete and missing modality settings. The code and benchmark will be publicly available at https://github.com/supertyd/FlexTrack/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。