用轻量级Mamba模型实现高效跨模态目标跟踪，兼顾速度与精度。
Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking
- 基于Vision Mamba设计统一框架，动态生成提示向量引导特征提取。
- 在COESOT、FE108和FELT V2数据集上均超越现有方法，精度提升显著。
- 适合追求实时性与鲁棒性的多模态视觉跟踪应用，如自动驾驶。
将传统RGB相机与类生物事件相机结合以实现鲁棒目标跟踪近年受到广泛关注。然而，多数现有跨模态跟踪算法依赖高复杂度的视觉变换器进行特征提取与模态融合，不仅带来巨大计算开销，也限制了跨模态交互效果。本文提出一种基于线性复杂度视觉Mamba网络的高效RGB-Event目标跟踪框架，命名为Mamba-FETrack V2。首先设计轻量级提示生成器，利用各模态嵌入特征与共享提示池动态生成特定模态可学习提示向量；随后，这些提示与模态特异性嵌入特征共同输入至基于Vision Mamba的FEMamba主干网络，实现提示引导的特征提取、跨模态交互与融合一体化处理；最后，融合表示送入跟踪头完成精准目标定位。在多个RGB-Event跟踪基准测试中，包括短期的COESOT数据集和长期的FE108与FELT V2数据集，实验结果表明该框架在性能与效率方面均表现优异。源代码与预训练模型将发布于https://github.com/Event-AHU/Mamba_FETrack。
原文摘要 · Abstract (English)
Combining traditional RGB cameras with bio-inspired event cameras for robust object tracking has garnered increasing attention in recent years. However, most existing multimodal tracking algorithms depend heavily on high-complexity Vision Transformer architectures for feature extraction and fusion across modalities. This not only leads to substantial computational overhead but also limits the effectiveness of cross-modal interactions. In this paper, we propose an efficient RGB-Event object tracking framework based on the linear-complexity Vision Mamba network, termed Mamba-FETrack V2. Specifically, we first design a lightweight Prompt Generator that utilizes embedded features from each modality, together with a shared prompt pool, to dynamically generate modality-specific learnable prompt vectors. These prompts, along with the modality-specific embedded features, are then fed into a Vision Mamba-based FEMamba backbone, which facilitates prompt-guided feature extraction, cross-modal interaction, and fusion in a unified manner. Finally, the fused representations are passed to the tracking head for accurate target localization. Extensive experimental evaluations on multiple RGB-Event tracking benchmarks, including short-term COESOT dataset and long-term datasets, i.e., FE108 and FELT V2, demonstrate the superior performance and efficiency of the proposed tracking framework. The source code and pre-trained models will be released on https://github.com/Event-AHU/Mamba_FETrack
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。