用扩散模型从草图生成3D点云，精准还原结构与位置。
DiffS-NOCS: 3D Point Cloud Reconstruction through Coloring Sketches to NOCS Maps Using Diffusion Models
- 草图转2D NOCS图，再拼合生成3D点云，避免直接3D重建难题。
- 在ShapeNet上实现高精度对齐草图的可控重建，保持多视角一致性。
- 适合需要草图引导3D生成的研究者，尤其关注结构控制与细节还原。
从条件草图重建3D点云极具挑战性。现有方法多直接在3D空间操作，但领域差异大且难以从2D草图准确恢复3D结构。理想模型还需融合草图与提示信息，面临多模态融合难题。本文提出DiffS-NOCS（基于扩散模型的草图到NOCS图），结合改进的多视角解码器与ControlNet，从草图生成包含3D结构和位置信息的2D NOCS图。通过融合多个视角的NOCS图重建3D点云。为增强草图理解，引入视点编码器提取视点特征；设计特征级多视角聚合网络作为去噪模块，促进跨视角信息交换，提升NOCS图生成中的3D一致性。在ShapeNet上的实验表明，DiffS-NOCS实现了与草图高度对齐、可控制且细粒度的点云重建。
原文摘要 · Abstract (English)
Reconstructing a 3D point cloud from a given conditional sketch is challenging. Existing methods often work directly in 3D space, but domain variability and difficulty in reconstructing accurate 3D structures from 2D sketches remain significant obstacles. Moreover, ideal models should also accept prompts for control, in addition with the sparse sketch, posing challenges in multi-modal fusion. We propose DiffS-NOCS (Diffusion-based Sketch-to-NOCS Map), which leverages ControlNet with a modified multi-view decoder to generate NOCS maps with embedded 3D structure and position information in 2D space from sketches. The 3D point cloud is reconstructed by combining multiple NOCS maps from different views. To enhance sketch understanding, we integrate a viewpoint encoder for extracting viewpoint features. Additionally, we design a feature-level multi-view aggregation network as the denoising module, facilitating cross-view information exchange and improving 3D consistency in NOCS map generation. Experiments on ShapeNet demonstrate that DiffS-NOCS achieves controllable and fine-grained point cloud reconstruction aligned with sketches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。