提出VeGaS,用折叠高斯建模视频动态,实现真实感编辑。
VeGaS: Video Gaussian Splatting
- 用折叠高斯分布建模视频非线性动态,通过条件2D高斯生成帧
- 在重建任务上优于当前最佳方法,支持自然的视频编辑操作
- 适合需要高质量视频重建与编辑的应用场景
隐式神经表示(INRs)利用神经网络将离散数据近似为连续函数。在视频领域,这类模型可将像素坐标与帧时间(或索引)映射为RGB值。尽管能有效压缩,但难以用于编辑。3D高斯溅射(3DGS)模型如视频高斯表示(VGR)可通过大量3D高斯编码视频,支持多种处理操作。然而其修改能力仅限于基础变换。为此,我们提出视频高斯溅射(VeGaS),实现真实感视频修改。核心是设计新型折叠高斯分布,捕捉视频流中的非线性动态,并通过条件2D高斯生成连续帧。实验表明,VeGaS在帧重建任务中超越现有最佳方案,且支持自然视频编辑。代码已开源:https://github.com/gmum/VeGaS。
原文摘要 · Abstract (English)
Implicit Neural Representations (INRs) employ neural networks to approximate discrete data as continuous functions. In the context of video data, such models can be utilized to transform the coordinates of pixel locations along with frame occurrence times (or indices) into RGB color values. Although INRs facilitate effective compression, they are unsuitable for editing purposes. One potential solution is to use a 3D Gaussian Splatting (3DGS) based model, such as the Video Gaussian Representation (VGR), which is capable of encoding video as a multitude of 3D Gaussians and is applicable for numerous video processing operations, including editing. Nevertheless, in this case, the capacity for modification is constrained to a limited set of basic transformations. To address this issue, we introduce the Video Gaussian Splatting (VeGaS) model, which enables realistic modifications of video data. To construct VeGaS, we propose a novel family of Folded-Gaussian distributions designed to capture nonlinear dynamics in a video stream and model consecutive frames by 2D Gaussians obtained as respective conditional distributions. Our experiments demonstrate that VeGaS outperforms state-of-the-art solutions in frame reconstruction tasks and allows realistic modifications of video data. The code is available at: https://github.com/gmum/VeGaS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。