直接在3D空间生成纹理颜色,避免2D映射的对齐与一致性问题。
NaTex: Seamless Texture Generation as Latent Color Diffusion
- 将纹理视为稠密颜色点云,用潜在色扩散模型直接生成3D纹理。
- 在纹理一致性和边界对齐上显著优于现有方法,尤其在遮挡区域表现更优。
- 支持零样本迁移,适用于材质生成、纹理修复等下游任务。
我们提出NaTex,一种原生纹理生成框架,直接在3D空间中预测纹理颜色。与依赖几何条件多视角扩散模型(MVD)合成2D多视图图像的先前方法不同,NaTex避免了MVD管道中的若干固有缺陷,包括难以处理需修补的遮挡区域、沿边界精确实现网格-纹理对齐,以及跨视图内容和颜色强度的一致性维持问题。NaTex采用新范式,将纹理视为密集颜色点云,提出潜在色扩散,包含一个几何感知的颜色点云变分自编码器(VAE)和一个多控制扩散变换器(DiT),完全基于3D数据从零训练。为实现精确对齐,引入原生几何控制,通过位置嵌入和几何潜在变量直接向DiT注入3D空间信息。我们共同设计了VAE-DiT架构,其中几何潜在变量由专用几何分支紧密耦合于颜色VAE提取,提供精细表面引导,保持强纹理对应关系。这些设计使NaTex展现出强大性能,在纹理一致性和对齐方面显著超越先前方法。此外,NaTex在无需训练或仅简单微调的情况下,也表现出强泛化能力,适用于多种下游应用,如材质生成、纹理精修、部件分割与纹理化。
原文摘要 · Abstract (English)
We present NaTex, a native texture generation framework that predicts texture color directly in 3D space. In contrast to previous approaches that rely on baking 2D multi-view images synthesized by geometry-conditioned Multi-View Diffusion models (MVDs), NaTex avoids several inherent limitations of the MVD pipeline. These include difficulties in handling occluded regions that require inpainting, achieving precise mesh-texture alignment along boundaries, and maintaining cross-view consistency and coherence in both content and color intensity. NaTex features a novel paradigm that addresses the aforementioned issues by viewing texture as a dense color point cloud. Driven by this idea, we propose latent color diffusion, which comprises a geometry-awared color point cloud VAE and a multi-control diffusion transformer (DiT), entirely trained from scratch using 3D data, for texture reconstruction and generation. To enable precise alignment, we introduce native geometry control that conditions the DiT on direct 3D spatial information via positional embeddings and geometry latents. We co-design the VAE-DiT architecture, where the geometry latents are extracted via a dedicated geometry branch tightly coupled with the color VAE, providing fine-grained surface guidance that maintains strong correspondence with the texture. With these designs, NaTex demonstrates strong performance, significantly outperforming previous methods in texture coherence and alignment. Moreover, NaTex also exhibits strong generalization capabilities, either training-free or with simple tuning, for various downstream applications, e.g., material generation, texture refinement, and part segmentation and texturing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。