arXiv:2410.19657cs.CV2024-10NeurIPS被引 63

用扩散模型生成可任意数量的3D高斯点,实现高效高质量渲染。

DiffGS: Functional Gaussian Splatting Diffusion

  • 将3D高斯点分解为概率、颜色、变换三个连续函数,实现可微生成。
  • 在多个任务上验证,能生成任意数量的高斯点,支持文本/图像/部分3D输入。
  • 结合八叉树采样优化,首次实现从函数直接提取高斯点,灵活性强。

3D高斯点阵(3DGS)在渲染速度和保真度方面表现优异,但其生成仍面临离散性和无结构性的挑战。本文提出DiffGS,一种基于潜在扩散模型的通用高斯生成器。核心思想是通过三个新函数对3DGS进行解耦建模:高斯概率、颜色与变换。通过这种解耦,将离散无结构的3DGS转化为连续高斯点函数,并训练潜在扩散模型以无条件或有条件方式生成这些函数。同时引入八叉树引导的采样与优化算法,可从生成函数中提取任意数量的高斯点。我们验证了DiffGS在无条件生成、文本/图像/部分3DGS条件生成以及点到高斯转换等多种任务中的有效性。DiffGS为灵活建模与生成3D高斯点提供了新方向。

原文摘要 · Abstract (English)

3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.

3D生成扩散模型高斯点阵

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。