首个基于脉冲神经网络的事件追踪框架,兼顾精度与能效。
SDTrack: A Baseline for Event-based Tracking via Spiking Neural Networks
- 用全局轨迹提示法融合事件流,增强时空表征
- Tiny版仅需19.61M参数、8.16mJ能耗,精度媲美主流方法
- 全脉冲驱动端到端设计,适合低功耗视觉应用
事件相机具备高时间分辨率、宽动态范围、低功耗和像素带宽优势。脉冲神经网络(SNN)通过离散脉冲信号天然适配事件数据,是事件追踪的理想选择。然而,现有结合人工神经网络(ANN)与SNN的方法存在架构不合理问题,影响能效与追踪性能。为此,我们提出首个基于Transformer的脉冲驱动追踪(SDTrack)流水线。其引入新型事件帧聚合方法——全局轨迹提示(GTP),有效捕捉全局轨迹信息并融合进事件流生成事件帧,增强时空表征。追踪器采用全脉冲驱动的SNN主干与轻量跟踪头。整个流程端到端运行,无需数据增强或后处理。大量实验表明,我们的SDTrack-Tiny在仅19.61M参数和8.16mJ能耗下实现竞争力精度;Base版本在三个数据集上达到当前最优表现。本工作为未来类脑视觉研究奠定坚实基础。
原文摘要 · Abstract (English)
Event cameras provide superior temporal resolution, dynamic range, energy efficiency, and pixel bandwidth. Spiking Neural Networks (SNNs) naturally complement event data through discrete spike signals, making them ideal for event-based tracking. However, current approaches combining Artificial Neural Networks (ANNs) and SNNs suffer from suboptimal architectures that compromise energy efficiency and limit tracking performance. To address these limitations, we propose the first Transformer-based \textbf{S}pike-\textbf{D}riven \textbf{T}racking (SDTrack) pipeline. It incorporates a novel event frame aggregation method called Global Trajectory Prompt (GTP) and a Transformer-based tracker. The GTP method effectively captures global trajectory information and aggregates it with event streams into event frames to enhance spatiotemporal representation. The Transformer-based tracker comprises a fully spike-driven SNN backbone and a simple tracking head. The SDTrack pipeline operates end-to-end without data augmentation or post-processing. Extensive experiments demonstrate that our SDTrack-Tiny pipeline achieves competitive accuracy with only 19.61$M$ parameters and 8.16$mJ$ energy consumption, while our Base version achieves state-of-the-art accuracy across three datasets. Our work establishes a solid foundation for future neuromorphic vision research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。