用2D图像监督训练3D高斯点云生成模型，无需3D真实数据。
A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision
- 用确定性教师模型生成噪声3D点云，实现2D监督下的3D扩散训练。
- 在物体和场景级数据上优于基线教师模型，生成更清晰、多样化的3D结构。
- 适合缺乏3D标注数据的3D生成任务，尤其适用于高保真点云建模。
我们提出一种新颖框架，仅使用2D监督训练3D图像条件扩散模型。从2D图像恢复3D结构本质是病态问题，存在多种可能重建解，生成模型因此成为自然选择。然而，现有3D生成模型多依赖完整3D监督，而大规模3D数据集稀缺，难以获取。为此，我们采用稀疏视图监督作为可扩展替代方案。尽管近期重建模型利用可微渲染结合稀疏视图监督将2D图像提升至3D，但其主要为确定性方法，无法捕捉多种合理解，在不确定区域产生模糊结果。训练3D扩散模型时面临的关键挑战在于：标准训练范式要求去噪过程与监督信号处于同一模态。我们通过解耦被去噪的噪声样本与监督信号，使前者保持3D，后者以2D形式提供。本方法利用确定性图像到3D模型（即“教师”）的次优预测生成噪声3D输入，从而在无需完整3D真值的情况下实现有效3D扩散训练。我们在物体级与场景级数据集上验证该框架，使用两种不同的3D Gaussian Splat (3DGS) 教师模型。结果表明，本方法持续优于这些确定性教师模型，证明了其在可扩展、高保真3D生成建模中的有效性。
原文摘要 · Abstract (English)
We present a novel framework for training 3D image-conditioned diffusion models using only 2D supervision. Recovering 3D structure from 2D images is inherently ill-posed due to the ambiguity of possible reconstructions, making generative models a natural choice. However, most existing 3D generative models rely on full 3D supervision, which is impractical due to the scarcity of large-scale 3D datasets. To address this, we propose leveraging sparse-view supervision as a scalable alternative. While recent reconstruction models use sparse-view supervision with differentiable rendering to lift 2D images to 3D, they are predominantly deterministic, failing to capture the diverse set of plausible solutions and producing blurry predictions in uncertain regions. A key challenge in training 3D diffusion models with 2D supervision is that the standard training paradigm requires both the denoising process and supervision to be in the same modality. We address this by decoupling the noisy samples being denoised from the supervision signal, allowing the former to remain in 3D while the latter is provided in 2D. Our approach leverages suboptimal predictions from a deterministic image-to-3D model-acting as a "teacher"-to generate noisy 3D inputs, enabling effective 3D diffusion training without requiring full 3D ground truth. We validate our framework on both object-level and scene-level datasets, using two different 3D Gaussian Splat (3DGS) teachers. Our results show that our approach consistently improves upon these deterministic teachers, demonstrating its effectiveness in scalable and high-fidelity 3D generative modeling. See our project page at https://lesson-in-splats.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。