用状态空间模型高效捕捉事件数据时空特征,实现低延迟光流估计。
Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow
- 设计时空状态空间模块,融合事件数据的时空相关性
- 推理速度比TMA快4.5倍,计算量减少8倍,性能媲美主流方法
- 适合高实时性要求的机器人、自动驾驶场景
事件相机开启了传统帧基相机无法企及的新可能,尤其在低延迟运动估计(光流)方面具有关键价值。此类应用对算法计算效率要求极高。尽管近年来基于CNN、RNN或ViT的深度学习方法表现优异,但通常计算开销大。相比之下,异步事件驱动方法如SNN和GNN虽高效,却难以充分捕捉所需的时空信息,制约性能提升。本文提出时空状态空间模型(STSSM)模块及新网络架构,构建一个计算极简且性能出色的解决方案。STSSM利用状态空间模型有效建模事件数据的时空相关性,在类似设置下相比ViT和基于CNN的架构表现出更高性能与更低复杂度。在DSEC基准上,本模型推理速度较TMA快4.5倍,计算量减少8倍;相较EV-FlowNet,计算量降低2倍,同时保持竞争力性能。代码将公开于https://github.com/AhmedHumais/E-STMFlow。
原文摘要 · Abstract (English)
Event cameras unlock new frontiers that were previously unthinkable with standard frame-based cameras. One notable example is low-latency motion estimation (optical flow), which is critical for many real-time applications. In such applications, the computational efficiency of algorithms is paramount. Although recent deep learning paradigms such as CNN, RNN, or ViT have shown remarkable performance, they often lack the desired computational efficiency. Conversely, asynchronous event-based methods including SNNs and GNNs are computationally efficient; however, these approaches fail to capture sufficient spatio-temporal information, a powerful feature required to achieve better performance for optical flow estimation. In this work, we introduce Spatio-Temporal State Space Model (STSSM) module along with a novel network architecture to develop an extremely efficient solution with competitive performance. Our STSSM module leverages state-space models to effectively capture spatio-temporal correlations in event data, offering higher performance with lower complexity compared to ViT, CNN-based architectures in similar settings. Our model achieves 4.5x faster inference and 8x lower computations compared to TMA and 2x lower computations compared to EV-FlowNet with competitive performance on the DSEC benchmark. Our code will be available at https://github.com/AhmedHumais/E-STMFlow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。