首个直接训练的脉冲神经网络多目标跟踪框架,实现低功耗实时追踪。
SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos
- 端到端训练脉冲网络,直接处理标准RGB视频流。
- 引入自适应归一化损失,提升小目标与密集场景定位精度。
- 适合低功耗嵌入式视觉系统,如无人机、智能摄像头。
类脑脉冲神经网络(SNN)在低功耗计算方面具有巨大潜力,但其在视觉任务中的应用仍主要局限于图像分类、目标检测和事件流追踪。相比之下,现实世界视觉系统广泛使用传统RGB视频流,而直接训练的SNN在复杂时序任务如多目标跟踪(MOT)上的潜力尚未充分探索。为此,我们提出SMTrack——首个针对标准RGB视频流的端到端训练深度脉冲神经网络多目标跟踪框架。SMTrack引入一种自适应且尺度感知的归一化Wasserstein距离损失(Asa-NWDLoss),通过在每个训练批次内计算平均物体尺寸并动态调整归一化因子,显著增强对小物体的敏感性。在关联阶段,集成TrackTrack身份模块以保持稳定一致的目标轨迹。在BEE24、MOT17、MOT20和DanceTrack数据集上的大量实验表明,SMTrack性能达到主流基于人工神经网络(ANN)的MOT方法水平,在复杂场景中实现了鲁棒且准确的SNN追踪。
原文摘要 · Abstract (English)
Brain-inspired Spiking Neural Networks (SNNs) exhibit significant potential for low-power computation, yet their application in visual tasks remains largely confined to image classification, object detection, and event-based tracking. In contrast, real-world vision systems still widely use conventional RGB video streams, where the potential of directly-trained SNNs for complex temporal tasks such as multi-object tracking (MOT) remains underexplored. To address this challenge, we propose SMTrack-the first directly trained deep SNN framework for end-to-end multi-object tracking on standard RGB videos. SMTrack introduces an adaptive and scale-aware Normalized Wasserstein Distance loss (Asa-NWDLoss) to improve detection and localization performance under varying object scales and densities. Specifically, the method computes the average object size within each training batch and dynamically adjusts the normalization factor, thereby enhancing sensitivity to small objects. For the association stage, we incorporate the TrackTrack identity module to maintain robust and consistent object trajectories. Extensive evaluations on BEE24, MOT17, MOT20, and DanceTrack show that SMTrack achieves performance on par with leading ANN-based MOT methods, advancing robust and accurate SNN-based tracking in complex scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。