用2D高斯点渲染实现超高效连续时空视频超分,推理速度几乎不随帧率提升而下降。
GS-STVSR: Ultra-Efficient Continuous Spatio-Temporal Video Super-Resolution via 2D Gaussian Splatting

- 基于2D高斯点渲染,避免密集网格查询,大幅降低计算开销。
- 在X32极端倍率下推理速度比现有方法快3倍以上,保持高质量输出。
- 适合需要实时处理高帧率视频的场景,如影视后期与直播增强。
连续时空视频超分辨率(C-STVSR)旨在通过任意缩放因子同时提升视频的空间分辨率和帧率,相比固定比例方法更具灵活性。近年来,基于隐式神经表示(INR)的方法在该任务上取得进展,但其依赖密集的像素级网格查询,导致计算成本随插值帧数线性增长,严重限制了推理效率。本文提出GS-STVSR,一种基于2D高斯点渲染(2D-GS)的超高效C-STVSR框架,通过连续运动建模驱动高斯核的时空演化,完全规避密集网格查询。我们利用协方差参数的强时序稳定性实现轻量中间拟合,设计光流引导的运动模块以在任意时间步推导高斯位置与颜色,引入协方差重采样对齐模块防止协方差漂移,并提出自适应偏移窗口应对大范围运动。在Vid4、GoPro和Adobe240上的大量实验表明,GS-STVSR在所有基准上均达到领先性能。尤其在常规时间尺度(X2–X8)下,其推理时间几乎恒定;在极端尺度X32下,推理速度提升超过3倍,展现出强大的实用性。
原文摘要 · Abstract (English)
Continuous Spatio-Temporal Video Super-Resolution (C-STVSR) aims to simultaneously enhance the spatial resolution and frame rate of videos by arbitrary scale factors, offering greater flexibility than fixed-scale methods that are constrained by predefined upsampling ratios. In recent years, methods based on Implicit Neural Representations (INR) have made significant progress in C-STVSR by learning continuous mappings from spatio-temporal coordinates to pixel values. However, these methods fundamentally rely on dense pixel-wise grid queries, causing computational cost to scale linearly with the number of interpolated frames and severely limiting inference efficiency. We propose GS-STVSR, an ultra-efficient C-STVSR framework based on 2D Gaussian Splatting (2D-GS) that drives the spatiotemporal evolution of Gaussian kernels through continuous motion modeling, bypassing dense grid queries entirely. We exploit the strong temporal stability of covariance parameters for lightweight intermediate fitting, design an optical flow-guided motion module to derive Gaussian position and color at arbitrary time steps, introduce a Covariance resampling alignment module to prevent covariance drift, and propose an adaptive offset window for large-scale motion. Extensive experiments on Vid4, GoPro, and Adobe240 show that GS-STVSR achieves state-of-the-art quality across all benchmarks. Moreover, its inference time remains nearly constant at conventional temporal scales (X2--X8) and delivers over X3 speedup at extreme scales X32, demonstrating strong practical applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。