用卷积高效捕捉时空模式,预测未来帧更准更快。
STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing
- 将时空维度合并后用单次卷积生成综合表征,避免传统方法依赖循环结构。
- 在多个数据集上达到顶尖性能,参数量和计算量大幅降低。
- 适合需要高效视频预测的场景,如自动驾驶、智能监控。
时空预测学习是一种自监督学习范式,通过基于历史帧预测未来帧来识别空间与时间模式。传统方法使用循环神经网络捕捉时间特征,虽有效但系统复杂度高、计算开销大。卷积虽更高效,却因对所有历史帧同等处理而难以刻画时序关系,且局部感受野限制了远距离帧间关联的捕捉能力。本文提出STLight,一种仅依赖通道卷积和深度卷积的全新时空学习方法。该方法通过重排时空维度,利用一次卷积融合空间与时间特征,生成统一的时空块表征。随后在纯卷积框架中同时关注邻近与远距离块间的交互,实现高效未来帧重建。模型在多个数据集的STL基准测试中均达最优表现,且显著降低参数量与计算浮点运算量(FLOPs)。代码已公开。
原文摘要 · Abstract (English)
Spatio-Temporal predictive Learning is a self-supervised learning paradigm that enables models to identify spatial and temporal patterns by predicting future frames based on past frames. Traditional methods, which use recurrent neural networks to capture temporal patterns, have proven their effectiveness but come with high system complexity and computational demand. Convolutions could offer a more efficient alternative but are limited by their characteristic of treating all previous frames equally, resulting in poor temporal characterization, and by their local receptive field, limiting the capacity to capture distant correlations among frames. In this paper, we propose STLight, a novel method for spatio-temporal learning that relies solely on channel-wise and depth-wise convolutions as learnable layers. STLight overcomes the limitations of traditional convolutional approaches by rearranging spatial and temporal dimensions together, using a single convolution to mix both types of features into a comprehensive spatio-temporal patch representation. This representation is then processed in a purely convolutional framework, capable of focusing simultaneously on the interaction among near and distant patches, and subsequently allowing for efficient reconstruction of the predicted frames. Our architecture achieves state-of-the-art performance on STL benchmarks across different datasets and settings, while significantly improving computational efficiency in terms of parameters and computational FLOPs. The code is publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。