用可学习的高斯点实现动态视频的自适应表示，兼顾时空效率与泛化能力。
Versatile Video Representation via Feed-Forward 2D Gaussian Splatting Tokenization
- 通过前馈式高斯点生成，按信息密度自适应分配空间编码资源。
- 在UCF101等数据集上实现视频重建与压缩的顶尖性能，动作识别也有提升。
- 适合需要高效视频建模的场景，如生成、压缩与跨任务迁移。
依赖固定网格和块状分块的现有视频表示方法普遍存在灵活性不足的问题。空间上，固定数量的令牌均匀分配常导致低信息区域过编码；时间上，缺乏对静态与动态内容的显式区分，难以有效减少冗余。本文提出基于前馈2D高斯点渲染（2DGS）的高斯视频变换器（GVT），首先通过提出的时空高斯嵌入（STGE）机制从前置视频片段中提取潜在刚性特征，并以2D高斯点集形式表示。这些高斯点在光栅化时根据信息密度自动调节渲染权重，增强空间适应性并避免每视频优化。为提升时间灵活性，引入高斯集分割（GSP）策略，将高斯点分为静态与动态两组，分别建模跨时间共享的静态内容和各时刻特异的动态内容，实现紧凑表达。我们在UCF101、Kinetics、DAVIS四个任务上评估，涵盖视频重建、动作识别、视频压缩与生成。结果表明，该方法在重建与压缩任务上达到当前最优，动作识别性能提升，生成效果接近基线MAGVIT-v2。
原文摘要 · Abstract (English)
Recent video representation methods that rely on fixed-grid, patch-wise tokenization often exhibit limited versatility.Spatially, uniformly allocating a fixed number of tokens often leads to over-encoding in low-information regions. Temporally, reducing redundancy remains challenging without explicitly distinguishing between static and dynamic content. In this work, we introduce the Gaussian Video Transformer (GVT), a versatile video representation framework built on a feed-forward 2D Gaussian Splatting (2DGS) tokenization scheme. We first extract latent rigid features from a video clip and represent them with a set of 2D Gaussians generated by our proposed Spatio-Temporal Gaussian Embedding (STGE) mechanism in a feed-forward manner. Such 2D Gaussians not only enhance spatial adaptability by assigning higher (resp., lower) rendering weights to regions with higher (resp., lower) information content during rasterization, but also improve generalization by avoiding per-video optimization. To enhance the temporal versatility, we introduce a Gaussian Set Partitioning (GSP) strategy that separates the 2D Gaussians into static and dynamic sets, which explicitly model static content shared across different time-steps and dynamic content specific to each time-step, enabling a compact representation. We evaluate GVT across four tasks: video reconstruction, video action recognition, video compression, and video generation, on the UCF101, Kinetics, and DAVIS datasets. The results demonstrate state-of-the-art reconstruction and compression performance, improved action recognition, and video generation performance comparable to the baseline MAGVIT-v2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。