首个基于ISTA适配器的混合神经网络跟踪模型,提升光流与图像融合效率。
ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking
- 用迭代收缩阈值算法设计适配器,打通ANN与SNN特征通道
- 在多个数据集上实现当前最佳追踪性能,能耗更低
- 适合做低功耗视觉追踪的工程师和研究员参考
RGB-Event追踪已成为利用彩色图像与动态脉冲事件互补优势的前沿方向。现有人工神经网络(ANN)难以充分挖掘事件流稀疏、异步的特性。近年来,结合ANN与脉冲神经网络(SNN)的混合架构被提出,但在跨模态、跨范式特征融合方面仍面临挑战。本文提出ISTASTrack,首个基于Transformer的ANN-SNN混合追踪框架,配备基于ISTA的适配器。双分支结构分别使用视觉Transformer提取RGB空间上下文,脉冲Transformer捕捉事件流的时空动态。为弥合ANN与SNN特征间的模态与范式差异,我们基于稀疏表示理论,通过展开迭代收缩阈值算法,系统设计了双向特征交互的模型级ISTA适配器。此外,在适配器中引入时间下采样注意力模块,使多步SNN特征与单步ANN特征在隐空间对齐,提升时序融合效果。在FE240hz、VisEvent、COESOT和FELT等基准测试中,ISTASTrack均达到当前最优性能,同时保持高能效,验证了混合架构在鲁棒视觉追踪中的有效性与实用性。代码已公开于https://github.com/lsying009/ISTASTrack.git。
原文摘要 · Abstract (English)
RGB-Event tracking has become a promising trend in visual object tracking to leverage the complementary strengths of both RGB images and dynamic spike events for improved performance. However, existing artificial neural networks (ANNs) struggle to fully exploit the sparse and asynchronous nature of event streams. Recent efforts toward hybrid architectures combining ANNs and spiking neural networks (SNNs) have emerged as a promising solution in RGB-Event perception, yet effectively fusing features across heterogeneous paradigms remains a challenge. In this work, we propose ISTASTrack, the first transformer-based \textbf{A}NN-\textbf{S}NN hybrid \textbf{Track}er equipped with \textbf{ISTA} adapters for RGB-Event tracking. The two-branch model employs a vision transformer to extract spatial context from RGB inputs and a spiking transformer to capture spatio-temporal dynamics from event streams. To bridge the modality and paradigm gap between ANN and SNN features, we systematically design a model-based ISTA adapter for bidirectional feature interaction between the two branches, derived from sparse representation theory by unfolding the iterative shrinkage thresholding algorithm. Additionally, we incorporate a temporal downsampling attention module within the adapter to align multi-step SNN features with single-step ANN features in the latent space, improving temporal fusion. Experimental results on RGB-Event tracking benchmarks, such as FE240hz, VisEvent, COESOT, and FELT, have demonstrated that ISTASTrack achieves state-of-the-art performance while maintaining high energy efficiency, highlighting the effectiveness and practicality of hybrid ANN-SNN designs for robust visual tracking. The code is publicly available at https://github.com/lsying009/ISTASTrack.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。