arXiv:2409.07454cs.CVcs.MM2024-09ECCV被引 23

用三角网格生成3D模型,让纹理更清晰、几何更准确。

DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation

  • 分两阶段:先粗调网格再精修纹理和形状
  • 多视角无调参使用2D扩散模型上色
  • 适合需要高质量纹理的3D生成场景

结合2D扩散模型与神经辐射场(NeRF)在文本到3D生成中广受欢迎,但其隐式3D表示缺乏对表面网格和纹理的显式建模,可能导致表面噪声大、纹理模糊或跨视角不一致。为此,我们提出DreamMesh,一种基于明确三角网格的新型文本到3D架构,可生成高保真显式3D模型。技术上,采用独特的粗到精方案:在粗阶段,通过文本引导的雅可比变换变形网格,并以多视角无调参方式利用2D扩散模型对网格进行纹理填充;在精阶段,联合优化网格形状与纹理贴图,生成高质量带纹理的三角网格。大量实验表明,DreamMesh显著优于当前最先进方法,在忠实还原3D内容、丰富纹理细节和提升几何质量方面表现优异。

原文摘要 · Abstract (English)

Learning radiance fields (NeRF) with powerful 2D diffusion models has garnered popularity for text-to-3D generation. Nevertheless, the implicit 3D representations of NeRF lack explicit modeling of meshes and textures over surfaces, and such surface-undefined way may suffer from the issues, e.g., noisy surfaces with ambiguous texture details or cross-view inconsistency. To alleviate this, we present DreamMesh, a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model. Technically, DreamMesh capitalizes on a distinctive coarse-to-fine scheme. In the coarse stage, the mesh is first deformed by text-guided Jacobians and then DreamMesh textures the mesh with an interlaced use of 2D diffusion models in a tuning free manner from multiple viewpoints. In the fine stage, DreamMesh jointly manipulates the mesh and refines the texture map, leading to high-quality triangle meshes with high-fidelity textured materials. Extensive experiments demonstrate that DreamMesh significantly outperforms state-of-the-art text-to-3D methods in faithfully generating 3D content with richer textual details and enhanced geometry. Our project page is available at https://dreammesh.github.io.

文本生成3D网格生成扩散模型纹理渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。