arXiv:2511.22896cs.CV2025-11

用扩散模型思想融合可见光与红外信息,提升多目标跟踪精度

DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking

  • 将跨模态融合设计为迭代对齐过程,通过双向引导对齐特征分布
  • 在VT-MOT上达41.7 HOTA,比现有方法提升1.54%相对性能
  • 适合需要鲁棒多模态感知的自动驾驶与机器人系统

多目标跟踪(MOT)是计算机视觉的基础任务,在自动驾驶和机器人中具有关键应用。融合可见光与热红外信息的多模态MOT对构建鲁棒自动驾驶系统尤为重要,但异构模态间特征分布差异大,简单拼接或相加难以有效融合,易引发模态冲突并降低跟踪精度。受扩散模型迭代优化机制启发,本文提出DM$^3$T,将多模态融合重构为迭代特征对齐过程,通过提出的跨模态扩散融合(C-MDF)模块实现双向引导,逐步将双模态特征投影至共享一致的特征流形上,从而学习互补信息并实现更深融合。此外,引入即插即用的扩散精炼器(DR)进一步优化统一特征表示,并设计分层跟踪器实现自适应置信度估计。该方法将检测、状态估计与数据关联统一于端到端在线框架中,无需复杂后处理。在VT-MOT基准上的大量实验表明,本方法取得41.7 HOTA,相较现有最优方法提升1.54%。代码与模型已开源。

原文摘要 · Abstract (English)

Multi-object tracking (MOT) is a fundamental task in computer vision with critical applications in autonomous driving and robotics. Multimodal MOT that integrates visible light and thermal infrared information is particularly essential for robust autonomous driving systems. However, effectively fusing these heterogeneous modalities is challenging. Simple strategies like concatenation or addition often fail to bridge the significant non-linear distribution gap between their feature representations, which can lead to modality conflicts and degrade tracking accuracy. Drawing inspiration from the connection between multimodal MOT and the iterative refinement in diffusion models, this paper proposes DM$^3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process to generate accurate and temporally coherent object trajectories. Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module. In this process, features from both modalities provide mutual guidance, iteratively projecting them onto a shared, consistent feature manifold. This enables the learning of complementary information and achieves deeper fusion compared to conventional methods. Additionally, we introduce a plug-and-play Diffusion Refiner (DR) to enhance and refine the unified feature representation. To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation. DM$^3$T unifies object detection, state estimation, and data association into a comprehensive online tracking framework without complex post-processing. Extensive experiments on the VT-MOT benchmark demonstrate that our method achieves 41.7 HOTA, representing a 1.54% relative improvement over existing state-of-the-art methods. The code and models are available at https://vranlee.github.io/DM-3-T/.

多目标跟踪多模态融合扩散模型自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。