arXiv:2411.00399cs.CVcs.GR2024-11中稿 · SIGGRAPH被引 21

用参考图生成匹配3D模型的风格化贴图,保持风格一致且不混入无关内容。

StyleTex: Style Image-Guided Texture Generation for 3D Models

  • 从参考图中分离风格与内容特征,通过CLIP嵌入减法实现
  • 生成贴图既保留参考图风格,又符合文本描述和模型几何细节
  • 适合游戏、影视等需要精准风格控制的3D内容创作

风格引导的纹理生成旨在根据参考风格图像和带有文本描述的3D网格,生成与之协调的纹理。尽管基于扩散模型的3D纹理生成方法(如蒸馏采样)在风格化游戏和电影中有广泛应用前景,但仍面临两大挑战:1)完全解耦参考图像中的风格与内容信息;2)使生成纹理与参考图像的颜色色调、风格及给定文本提示对齐。为此,本文提出StyleTex,一种基于扩散模型的3D风格纹理生成框架。核心思想是在蒸馏采样中忽略内容,仅保留风格信息。具体而言,给定参考图像,首先通过从图像CLIP嵌入中减去其在内容特征方向(由文本CLIP嵌入表示)的正交投影,分离出风格特征。该创新的解耦机制可生成独立的风格与内容特征。随后将风格特征注入交叉注意力模块以融入生成过程,并利用内容特征作为负提示进一步消除内容干扰。最终将上述策略整合至StyleTex,生成的纹理不仅保留参考图风格,还契合文本提示和输入3D网格的内在结构。定量与定性实验表明,本方法显著优于现有基线方法。

原文摘要 · Abstract (English)

Style-guided texture generation aims to generate a texture that is harmonious with both the style of the reference image and the geometry of the input mesh, given a reference style image and a 3D mesh with its text description. Although diffusion-based 3D texture generation methods, such as distillation sampling, have numerous promising applications in stylized games and films, it requires addressing two challenges: 1) decouple style and content completely from the reference image for 3D models, and 2) align the generated texture with the color tone, style of the reference image, and the given text prompt. To this end, we introduce StyleTex, an innovative diffusion-model-based framework for creating stylized textures for 3D models. Our key insight is to decouple style information from the reference image while disregarding content in diffusion-based distillation sampling. Specifically, given a reference image, we first decompose its style feature from the image CLIP embedding by subtracting the embedding's orthogonal projection in the direction of the content feature, which is represented by a text CLIP embedding. Our novel approach to disentangling the reference image's style and content information allows us to generate distinct style and content features. We then inject the style feature into the cross-attention mechanism to incorporate it into the generation process, while utilizing the content feature as a negative prompt to further dissociate content information. Finally, we incorporate these strategies into StyleTex to obtain stylized textures. The resulting textures generated by StyleTex retain the style of the reference image, while also aligning with the text prompts and intrinsic details of the given 3D mesh. Quantitative and qualitative experiments show that our method outperforms existing baseline methods by a significant margin.

3D生成风格迁移扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。