用预训练生成模型统一追踪多模态图像,提升复杂场景稳定性。
Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking
- 用Stable Diffusion的UNet提取多模态特征,实现统一追踪
- 在TNL2K数据集上AUC比OneTracker高8.3%
- 适合需要多模态融合的视觉追踪研究者
多模态目标追踪通过融合深度、热红外、事件流和语言等辅助模态,提升复杂场景下的追踪稳定性。现有方法通常以RGB追踪器为基础,仅从训练数据中学习辅助模态理解,受限于多模态训练数据稀缺,性能不佳。为此,本文提出统一多模态追踪器Diff-MM,利用预训练文本到图像生成模型的多模态理解能力。Diff-MM通过设计并行特征提取管道,将Stable Diffusion的UNet用于特征提取,支持成对图像输入进行目标追踪,并引入多模态子模块微调方法,学习不同模态间的互补信息。借助生成模型中的丰富先验知识,实现了适用于RGB-N/D/T/E追踪的统一参数追踪器。实验表明,该方法在近期追踪器中表现优异,例如在TNL2K数据集上AUC优于OneTracker 8.3%。
原文摘要 · Abstract (English)
Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language to provide additional information beyond RGB images, showing great potential in improving tracking stabilization in complex scenarios. Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data. Constrained by the limited multi-modal training data, the performance of these methods is unsatisfactory. To alleviate this limitation, this work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model. Diff-MM leverages the UNet of pre-trained Stable Diffusion as a tracking feature extractor through the proposed parallel feature extraction pipeline, which enables pairwise image inputs for object tracking. We further introduce a multi-modal sub-module tuning method that learns to gain complementary information between different modalities. By harnessing the extensive prior knowledge in the generation model, we achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking. Experimental results demonstrate the promising performance of our method compared with recently proposed trackers, e.g., its AUC outperforms OneTracker by 8.3% on TNL2K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。