arXiv:2503.19721cs.CV2025-03AAAI被引 12

EventMamba提升事件视频重建的时空局部性,速度更快、画质更好。

EventMamba: Enhancing Spatio-Temporal Locality with State Space Models for Event-Based Video Reconstruction

  • 用随机窗口偏移替代固定分区,增强空间局部连接
  • 设计一致遍历序列化,保持时空邻近事件的连续性
  • 比基于Transformer的方法快得多,重建质量更优

利用Mamba强大的线性全局建模能力,其在计算机视觉中表现突出。然而,现有基于Mamba的视觉模型忽视了事件驱动任务的细节,尤其在事件视频重建(EBVR)中。EBVR要求空间平移不变性,并关注时空域内的局部事件关系。传统Mamba算法采用静态窗口划分和标准重塑扫描方式,导致局部连接性严重丢失。为此,我们提出EventMamba——专为EBVR任务设计的模型。EventMamba通过在空间域引入随机窗口偏移(RWO),摆脱固定分区限制;同时采用新的时空一致遍历序列化方法,保持相邻事件在时空上的邻近性。这些改进使EventMamba在保留Mamba强大建模能力的同时,显著增强事件数据的时空局部性。多数据集全面测试表明,EventMamba显著提升视频重建效果,计算速度大幅加快,视觉质量优于基于Transformer的方法。

原文摘要 · Abstract (English)

Leveraging its robust linear global modeling capability, Mamba has notably excelled in computer vision. Despite its success, existing Mamba-based vision models have overlooked the nuances of event-driven tasks, especially in video reconstruction. Event-based video reconstruction (EBVR) demands spatial translation invariance and close attention to local event relationships in the spatio-temporal domain. Unfortunately, conventional Mamba algorithms apply static window partitions and standard reshape scanning methods, leading to significant losses in local connectivity. To overcome these limitations, we introduce EventMamba--a specialized model designed for EBVR tasks. EventMamba innovates by incorporating random window offset (RWO) in the spatial domain, moving away from the restrictive fixed partitioning. Additionally, it features a new consistent traversal serialization approach in the spatio-temporal domain, which maintains the proximity of adjacent events both spatially and temporally. These enhancements enable EventMamba to retain Mamba's robust modeling capabilities while significantly preserving the spatio-temporal locality of event data. Comprehensive testing on multiple datasets shows that EventMamba markedly enhances video reconstruction, drastically improving computation speed while delivering superior visual quality compared to Transformer-based methods.

事件视频状态空间模型时空建模Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。