用轨迹感知的Mamba模型,高效实现在线视频超分。
Trajectory-aware Shifted State Space Models for Online Video Super-Resolution
- 基于轨迹选择前帧关键信息,结合移位SSM聚合时序特征。
- 在三个数据集上超越6个基准模型,计算量减少超22.7%。
- 适合需要低延迟、高时效性的实时视频增强场景。
在线视频超分辨率(VSR)旨在利用历史帧恢复当前高清帧,广泛应用于真实场景。现有方法多仅依赖单个前帧进行对齐,难以建模长时依赖。本文提出轨迹感知移位状态空间模型(TS-Mamba),结合长程轨迹建模与低复杂度Mamba结构,实现高效的时空信息聚合。首先构建视频内运动轨迹,筛选最相似的前帧令牌;随后通过轨迹感知移位Mamba聚合模块(TSMA)整合信息。该模块基于希尔伯特扫描与相应偏移操作,弥补扫描损失并增强空间连续性。此外,设计轨迹感知损失函数以监督轨迹生成,提升令牌选择精度。在三个主流VSR数据集上的实验表明,相比六种基准模型,TS-Mamba在多数情况下达到最优性能,并实现超过22.7%的计算量降低(以MACs计)。
原文摘要 · Abstract (English)
Online video super-resolution (VSR) is an important technique for many real-world video processing applications, which aims to restore the current high-resolution video frame based on temporally previous frames. Most of the existing online VSR methods solely employ one neighboring previous frame to achieve temporal alignment, which limits long-range temporal modeling of videos. Recently, state space models (SSMs) have been proposed with linear computational complexity and a global receptive field, which significantly improve computational efficiency and performance. In this context, this paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba), leveraging both long-term trajectory modeling and low-complexity Mamba to achieve efficient spatio-temporal information aggregation. Specifically, TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames. Then, a Trajectory-aware Shifted Mamba Aggregation (TSMA) module consisting of proposed shifted SSMs blocks is employed to aggregate the selected tokens. The shifted SSMs blocks are designed based on Hilbert scannings and corresponding shift operations to compensate for scanning losses and strengthen the spatial continuity of Mamba. Additionally, we propose a trajectory-aware loss function to supervise the trajectory generation, ensuring the accuracy of token selection when training our model. Extensive experiments on three widely used VSR test datasets demonstrate that compared with six online VSR benchmark models, our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% complexity reduction (in MACs).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。