用文本生成3D网格序列的多视角连贯纹理,无需额外训练。
Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models
- 通过UV空间潜变量聚合同步多视角扩散过程,提升一致性。
- 在未贴图网格序列上生成多帧连贯视频,避免模糊问题。
- 适合影视游戏中的自动材质生成,无需专业艺术家干预。
3D网格在动画和低内存占用方面具有优势,广泛应用于电影、游戏、AR与VR中。然而,为网格序列创建时间一致且逼真的纹理仍需大量人工工作。尽管视频扩散模型擅长文本驱动的视频生成,却缺乏对3D几何结构的感知,难以实现3D网格的多视角一致纹理生成。本文提出Tex4D,一种零样本方法,将网格序列的固有3D几何信息与视频扩散模型的表达能力结合,生成多视角与时间一致的4D纹理。输入为无纹理网格序列与文本提示,通过在UV空间进行潜变量聚合,同步不同视角的扩散过程以增强多视角一致性;利用条件视频生成模型的先验知识保障时间一致性。针对直接融合导致的模糊问题,分析其成因并提出改进的DDIM采样策略;同时引入参考潜变量,强化去噪过程中帧间关联。据我们所知,Tex4D是首个专为4D场景纹理化设计的方法。大量实验表明,其在基于无纹理网格序列生成多视角、多帧一致视频方面表现卓越。
原文摘要 · Abstract (English)
3D meshes are widely used in computer vision and graphics for their efficiency in animation and minimal memory use, playing a crucial role in movies, games, AR, and VR. However, creating temporally consistent and realistic textures for mesh sequences remains labor-intensive for professional artists. On the other hand, while video diffusion models excel at text-driven video generation, they often lack 3D geometry awareness and struggle with achieving multi-view consistent texturing for 3D meshes. In this work, we present Tex4D, a zero-shot approach that integrates inherent 3D geometry knowledge from mesh sequences with the expressiveness of video diffusion models to produce multi-view and temporally consistent 4D textures. Given an untextured mesh sequence and a text prompt as inputs, our method enhances multi-view consistency by synchronizing the diffusion process across different views through latent aggregation in the UV space. To ensure temporal consistency, we leverage prior knowledge from a conditional video generation model for texture synthesis. However, straightforwardly combining the video diffusion model and the UV texture aggregation leads to blurry results. We analyze the underlying causes and propose a simple yet effective modification to the DDIM sampling process to address this issue. Additionally, we introduce a reference latent texture to strengthen the correlation between frames during the denoising process. To the best of our knowledge, Tex4D is the first method specifically designed for 4D scene texturing. Extensive experiments demonstrate its superiority in producing multi-view and multi-frame consistent videos based on untextured mesh sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。