4DSTR通过时空校正提升动态内容生成质量与一致性。
4DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generation
- 引入时空校正机制，优化4D高斯点的形变与旋转。
- 在视频转4D生成任务中达到顶尖性能，保持高质量与一致性。
- 适合需要高精度动态3D内容生成的研究与应用者。
近期2D图像和3D形状生成的显著进展推动了动态4D内容生成的关注。然而，现有4D生成方法普遍存在时空不一致问题，且难以适应快速时间变化，主要由于缺乏有效的时空建模。为此，我们提出一种新型4D生成网络4DSTR，通过时空校正调制生成式4D高斯点阵（Gaussian Splatting）。具体而言，设计了跨时序的时序相关性以校正可变形的尺度与旋转，保障时间一致性。此外，提出自适应空间稠密化与剪枝策略，通过感知前一帧运动动态增删高斯点，有效应对显著的时间变化。大量实验表明，4DSTR在视频转4D生成任务中表现领先，显著提升重建质量、时空一致性及对快速运动的适应能力。
原文摘要 · Abstract (English)
Remarkable advances in recent 2D image and 3D shape generation have induced a significant focus on dynamic 4D content generation. However, previous 4D generation methods commonly struggle to maintain spatial-temporal consistency and adapt poorly to rapid temporal variations, due to the lack of effective spatial-temporal modeling. To address these problems, we propose a novel 4D generation network called 4DSTR, which modulates generative 4D Gaussian Splatting with spatial-temporal rectification. Specifically, temporal correlation across generated 4D sequences is designed to rectify deformable scales and rotations and guarantee temporal consistency. Furthermore, an adaptive spatial densification and pruning strategy is proposed to address significant temporal variations by dynamically adding or deleting Gaussian points with the awareness of their pre-frame movements. Extensive experiments demonstrate that our 4DSTR achieves state-of-the-art performance in video-to-4D generation, excelling in reconstruction quality, spatial-temporal consistency, and adaptation to rapid temporal movements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。