用点云先验控制3D生成，让模型更懂几何结构。
Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors
- 用点云作为几何先验，替代随机初始化，引导生成结构
- 在物体和场景上均超越现有方法，渲染质量和结构保真度更高
- 适合需要精准几何控制的3D内容创作场景
当前3D生成主要依赖图像或文本条件，而易获取的点云先验仍被忽视。实际中，通过激光雷达或前向预测器（如VGGT）可轻松获得可见区域点云，提供明确几何约束，但现有方法未能有效利用。本文提出Points-to-3D，基于扩散模型TRELLIS，将点云先验融入潜空间初始化，取代纯噪声初始化。设计结构补全网络，在任务特定数据上训练，结合分阶段采样策略（先结构补全，再边界细化），在保留输入可见区域的同时完成全局几何生成。实验表明，该方法在物体与场景生成任务中，均显著优于当前最优基线，在渲染质量与几何保真度方面表现优异，证明显式嵌入点云先验对实现更精确、可控制的3D生成的有效性。
原文摘要 · Abstract (English)
Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input priors. In practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation. Project page: https://jiatongxia.github.io/points2-3D/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。