将多视角扩散模型蒸馏为3D生成器，实现快速高保真3D内容生成。
Distilling Multi-view Diffusion Models into 3D Generators
- 通过模拟微分方程轨迹，融合多视角视觉与几何知识
- 单图转3D高斯点云仅需0.06秒，生成质量优于纯3D训练模型
- 适合需要高效3D生成的场景，如游戏、虚拟现实开发
我们提出DD3G，一种将多视角扩散模型（MV-DM）蒸馏为3D生成器的方法，采用高斯点阵实现。该方法通过模拟教师模型的常微分方程（ODE）轨迹，压缩并整合丰富的视觉与空间几何知识，使学生生成器泛化能力优于仅依赖3D数据训练的模型。不同于以往的近似优化方法，我们对齐教师与学生表示空间，将概率流直接传递，避免因概率采样导致的目标不一致问题。为应对概率流与3D高斯属性耦合带来的挑战，我们设计了包含模式提取与渐进解码的PEPD生成器，可在0.06秒内将单张图像转换为3D高斯点云。此外，为减少知识损失并克服稀疏视图监督，我们引入联合优化目标，通过显式监督与隐式验证保障生成质量。利用现有2D生成模型，我们构建了120,000张高质量RGBA图像用于蒸馏。在合成及公开数据集上的实验验证了方法的有效性。项目地址：https://qinbaigao.github.io/DD3G_project/
原文摘要 · Abstract (English)
We introduce DD3G, a formulation that Distills a multi-view Diffusion model (MV-DM) into a 3D Generator using gaussian splatting. DD3G compresses and integrates extensive visual and spatial geometric knowledge from the MV-DM by simulating its ordinary differential equation (ODE) trajectory, ensuring the distilled generator generalizes better than those trained solely on 3D data. Unlike previous amortized optimization approaches, we align the MV-DM and 3D generator representation spaces to transfer the teacher's probabilistic flow to the student, thus avoiding inconsistencies in optimization objectives caused by probabilistic sampling. The introduction of probabilistic flow and the coupling of various attributes in 3D Gaussians introduce challenges in the generation process. To tackle this, we propose PEPD, a generator consisting of Pattern Extraction and Progressive Decoding phases, which enables efficient fusion of probabilistic flow and converts a single image into 3D Gaussians within 0.06 seconds. Furthermore, to reduce knowledge loss and overcome sparse-view supervision, we design a joint optimization objective that ensures the quality of generated samples through explicit supervision and implicit verification. Leveraging existing 2D generation models, we compile 120k high-quality RGBA images for distillation. Experiments on synthetic and public datasets demonstrate the effectiveness of our method. Our project is available at: https://qinbaigao.github.io/DD3G_project/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。