用轻量分域注意力提升视频理解，性能优于同类模型。
VideoSEMA: a scalable and efficient Mamba-like attention for video understanding
- 分空间-时间设计：空间用类Mamba局部+全局注意力，时间用softmax注意力。
- 在K400上超越更重的ViT和Mamba模型，在SSv2上同规模最佳。
- 高分辨率下鲁棒性强，适合扩展到长视频任务。
我们提出面向视频理解（分类）的Split Space-Time Attention模型VideoSEMA，由空间的可扩展高效Mamba-like注意力（SEMA）块与时间的softmax注意力组成。在每帧中，SEMA注意力在Mamba宏观架构下并行执行局部窗口注意力与全局平均操作。在特定秩条件下，证明其计算更优的分域空间-时间注意力等价于全空间-时间注意力。在K400基准上，VideoSEMA优于更重的视觉变压器与Mamba模型；在SSv2上，同参数规模下取得最高准确率。当图像分辨率从标准的$224^2$升至$1024^2$且无需微调时，VideoSEMA在K400上的准确率下降远小于VideoMamba。未来有望通过稀疏/膨胀时间注意力扩展至更长视频。
原文摘要 · Abstract (English)
We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard $224^2$ to $1024^2$ on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。