针对电影拍摄稀疏摄像头场景，实现高质量动态3D重建。
Splatography: Sparse multi-view dynamic Gaussian Splatting for filmmaking challenges
- 分离前景与背景的高斯点云，分轨学习动态变化
- 在3D场景中提升3分贝PSNR，模型体积减半
- 无需密集标注，支持透明与动态材质的分割重建
可变形高斯点阵（Deformable Gaussian Splatting, GS）通过学习对初始高斯表示进行形变，实现了从密集多视角视频（MVV）中进行逼真动态3D重建。然而，在电影制作中受限于预算导致的稀疏相机配置，现有最先进（SotA）方法在捕捉复杂动态特征时表现受限。为此，我们提出一种新方法：利用t=0时刻的稀疏掩码将初始高斯点与形变场分为前景和背景两部分，并分别在不同损失函数下进行初始预训练。在动态训练阶段，根据常见电影制作实践，对两部分采用不同的参数建模：前景包含丰富动态特征，学习颜色、位置与旋转变化；背景（如剧组人员与设备）通常较暗且动态少，仅学习点位移变化。在3D与2.5D娱乐数据集上的实验表明，该方法在定性和定量指标上均达到最先进水平，3D场景中最高提升3 PSNR，模型规模减半。与最先进方法相比，本方法无需密集掩码监督，即可生成包含透明与动态纹理的分割式动态重建。代码与视频对比已公开：https://azzarelli.github.io/splatographypage/index.html
原文摘要 · Abstract (English)
Deformable Gaussian Splatting (GS) accomplishes photorealistic dynamic 3-D reconstruction from dense multi-view video (MVV) by learning to deform a canonical GS representation. However, in filmmaking, tight budgets can result in sparse camera configurations, which limits state-of-the-art (SotA) methods when capturing complex dynamic features. To address this issue, we introduce an approach that splits the canonical Gaussians and deformation field into foreground and background components using a sparse set of masks for frames at t=0. Each representation is separately trained on different loss functions during canonical pre-training. Then, during dynamic training, different parameters are modeled for each deformation field following common filmmaking practices. The foreground stage contains diverse dynamic features so changes in color, position and rotation are learned. While, the background containing film-crew and equipment, is typically dimmer and less dynamic so only changes in point position are learned. Experiments on 3-D and 2.5-D entertainment datasets show that our method produces SotA qualitative and quantitative results; up to 3 PSNR higher with half the model size on 3-D scenes. Unlike the SotA and without the need for dense mask supervision, our method also produces segmented dynamic reconstructions including transparent and dynamic textures. Code and video comparisons are available online: https://azzarelli.github.io/splatographypage/index.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。