分步生成4D视频，先布局后细节，效果更真实。
4DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation
- 先生成多视角粗略布局，再基于结构先验精细生成
- 在16个视角、每视角21帧的动态3D数据上达到领先性能
- 适合需要高精度4D内容生成的研究者与开发者
针对直接生成高维数据（如4D）的复杂性，本文提出4DVD——一种级联式视频扩散模型，通过解耦方式生成4D内容。不同于以往同时建模3D空间与时间特征的方法，4DVD将任务分解为两个子过程：粗粒度多视角布局生成与结构感知条件生成，并有效融合。给定单目视频，4DVD首先生成具有优异跨视角与时间一致性的密集视图布局；随后利用该布局先验，结合输入视频的精细外观信息，生成高质量密集视图视频。得益于这一设计，可精准优化显式4D表示（如4D Gaussian），拓展实际应用。为训练模型，我们从Objaverse基准构建动态3D对象数据集D-Objaverse，为每个物体渲染16个视角、共21帧的视频。大量实验表明，4DVD在新视角合成与4D生成任务上均达到当前最优表现。
原文摘要 · Abstract (English)
Given the high complexity of directly generating high-dimensional data such as 4D, we present 4DVD, a cascaded video diffusion model that generates 4D content in a decoupled manner. Unlike previous multi-view video methods that directly model 3D space and temporal features simultaneously with stacked cross view/temporal attention modules, 4DVD decouples this into two subtasks: coarse multi-view layout generation and structure-aware conditional generation, and effectively unifies them. Specifically, given a monocular video, 4DVD first predicts the dense view content of its layout with superior cross-view and temporal consistency. Based on the produced layout priors, a structure-aware spatio-temporal generation branch is developed, combining these coarse structural priors with the exquisite appearance content of input monocular video to generate final high-quality dense-view videos. Benefit from this, explicit 4D representation~(such as 4D Gaussian) can be optimized accurately, enabling wider practical application. To train 4DVD, we collect a dynamic 3D object dataset, called D-Objaverse, from the Objaverse benchmark and render 16 videos with 21 frames for each object. Extensive experiments demonstrate our state-of-the-art performance on both novel view synthesis and 4D generation. Our project page is https://4dvd.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。