用多视角语义生成一致且细节丰富的3D模型
Shape from Semantics: 3D Shape Generation from Multi-View Semantics
- 通过解耦几何与外观，利用多视角法向-深度扩散先验补全形状
- 生成具有精细结构、连贯纹理和视角平滑过渡的高质量3D网格
- 适合需要语义驱动3D设计的创作者和工业建模场景
现有3D重建方法依赖2D图像、点云、轮廓或单一语义引导，限制了3D建模的创造性。本文提出新任务「由语义生成形状」，旨在根据多视角语义生成几何与外观一致的3D模型。所生成模型包含多个语义元素，易于观察者区分。采用生成模型作为先验，解耦几何与外观关联。提出局部几何感知蒸馏（LGAD）策略，利用多视角法向-深度扩散先验补全部分几何，确保形状真实。引入视图自适应引导尺度，实现跨视角语义平滑过渡。外观建模采用物理渲染生成高质量材质属性，并烘焙至可制造网格。大量实验表明，本方法能生成结构良好、细节丰富、纹理连贯且过渡平滑的视觉吸引人3D形状。项目页面：https://shapefromsemantics.github.io
原文摘要 · Abstract (English)
Existing 3D reconstruction methods utilize guidances such as 2D images, 3D point clouds, shape contours and single semantics to recover the 3D surface, which limits the creative exploration of 3D modeling. In this paper, we propose a novel 3D modeling task called ``Shape from Semantics'', which aims to create 3D models whose geometry and appearance are consistent with the given text semantics when viewed from different views. The reconstructed 3D models incorporate more than one semantic elements and are easy for observers to distinguish. We adopt generative models as priors and disentangle the connection between geometry and appearance to solve this challenging problem. Specifically, we propose Local Geometry-Aware Distillation (LGAD), a strategy that employs multi-view normal-depth diffusion priors to complete partial geometries, ensuring realistic shape generation. We also integrate view-adaptive guidance scales to enable smooth semantic transitions across views. For appearance modeling, we adopt physically based rendering to generate high-quality material properties, which are subsequently baked into fabricable meshes. Extensive experimental results demonstrate that our method can generate meshes with well-structured, intricately detailed geometries, coherent textures, and smooth transitions, resulting in visually appealing 3D shape designs. Project page: https://shapefromsemantics.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。