通过解耦时空特征迁移，提升4D生成质量与一致性。
Orthogonal Spatial-temporal Distributional Transfer for 4D Generation
- 解耦空间与时间潜变量，分别从3D和视频扩散模型迁移先验。
- 在公开数据集上实现比现有方法更高的时空一致性与生成质量。
- 适合关注4D内容生成、时空建模的科研与工业开发者。
在AIGC时代，高质量4D内容生成受到越来越多关注。然而，当前4D合成研究严重受限于缺乏大规模4D数据集，导致模型难以充分学习关键的时空特征，阻碍了该领域进展。为此，我们提出一种新框架，将现有3D扩散模型中的丰富空间先验和视频扩散模型中的时间先验迁移至4D合成。构建了时空解耦的4D（STD-4D）扩散模型，通过解耦的空间与时间潜变量生成4D感知视频。为优化特征迁移，设计了正交时空分布转移（Orster）机制，对时空特征分布进行建模并注入模型。同时，在4D构建过程中，提出时空感知的HexPlane（ST-HexPlane），融合迁移的时空特征，提升4D形变与4D高斯特征建模能力。实验表明，该方法显著优于现有方法，在时空一致性和4D生成质量上均表现更优。
原文摘要 · Abstract (English)
In the AIGC era, generating high-quality 4D content has garnered increasing research attention. Unfortunately, current 4D synthesis research is severely constrained by the lack of large-scale 4D datasets, preventing models from adequately learning the critical spatial-temporal features necessary for high-quality 4D generation, thus hindering progress in this domain. To combat this, we propose a novel framework that transfers rich spatial priors from existing 3D diffusion models and temporal priors from video diffusion models to enhance 4D synthesis. We develop a spatial-temporal-disentangled 4D (STD-4D) Diffusion model, which synthesizes 4D-aware videos through disentangled spatial and temporal latents. To facilitate the best feature transfer, we design a novel Orthogonal Spatial-temporal Distributional Transfer (Orster) mechanism, where the spatiotemporal feature distributions are carefully modeled and injected into the STD-4D Diffusion. Furthermore, during the 4D construction, we devise a spatial-temporal-aware HexPlane (ST-HexPlane) to integrate the transferred spatiotemporal features, thereby improving 4D deformation and 4D Gaussian feature modeling. Experiments demonstrate that our method significantly outperforms existing approaches, achieving superior spatial-temporal consistency and higher-quality 4D synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。