TRecViT用时-空-通道分解，实现低开销实时视频建模。
TRecViT: A Recurrent Video Transformer
- 分维度设计：时序用门控循环单元，空间用自注意力，通道用MLP
- 参数少3倍、内存小12倍、算力低5倍，推理达300帧/秒
- 首个状态空间模型类因果视频模型，适合实时应用与高效部署
我们提出一种新的因果视频建模模块，基于时间-空间-通道的因子分解结构：门控线性循环单元（LRUs）处理时序信息混合，自注意力层处理空间混合，MLP处理通道混合。由此构建的TRecViT架构具有因果性，在稀疏和密集任务上表现优异，支持监督或自监督训练，是首个属于状态空间模型家族的因果视频模型。在大规模视频数据集SSv2和Kinetics400上，其性能优于或持平主流非因果模型ViViT-L，同时参数量仅为后者的1/3，内存占用减少12倍，计算量降低5倍，推理吞吐量约300帧/秒，可流畅实现实时处理。相比因果变压器模型（TSM、RViT）及其他循环模型（如LSTM），TRecViT在挑战性的SSv2数据集上达到最先进水平。代码与模型权重已公开于https://github.com/google-deepmind/trecvit。
原文摘要 · Abstract (English)
We propose a novel block for \emph{causal} video modelling. It relies on a time-space-channel factorisation with dedicated blocks for each dimension: gated linear recurrent units (LRUs) perform information mixing over time, self-attention layers perform mixing over space, and MLPs over channels. The resulting architecture \emph{TRecViT} is causal and shows strong performance on sparse and dense tasks, trained in supervised or self-supervised regimes, being the first causal video model in the state-space models family. Notably, our model outperforms or is on par with the popular (non-causal) ViViT-L model on large scale video datasets (SSv2, Kinetics400), while having $3\times$ less parameters, $12\times$ smaller memory footprint, and $5\times$ lower FLOPs count than the full self-attention ViViT, with an inference throughput of about 300 frames per second, running comfortably in real-time. When compared with causal transformer-based models (TSM, RViT) and other recurrent models like LSTM, TRecViT obtains state-of-the-art results on the challenging SSv2 dataset. Code and checkpoints are available online https://github.com/google-deepmind/trecvit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。