从单图或多图生成高质量3D形状与纹理,支持多种输入格式。
Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and Texture Generation
- 用变分自编码器和扩散模型生成3D形状,支持文本、单图、多图输入。
- 分阶段生成纹理,通过一致性调度器确保多视角纹理无缝衔接。
- 适合需要高效生成逼真3D内容的开发者或创作者使用。
本文提出一个综合性框架,可从单张图像、多视角图像或文本描述等多样化输入中生成高质量3D形状与纹理。3D形状生成采用变分自编码器(VAE)将隐式几何编码至潜在空间,并利用扩散网络在输入提示条件下生成潜在表示,同时改进结构以增强模型容量;另探索了艺术家创建网格(AM)生成方法,在简单几何上表现良好。纹理生成包含多阶段流程:先生成正面图像,再生成多视角图像,继而进行RGB到PBR材质转换,最后通过高分辨率多视角纹理精修。每阶段均引入一致性调度器,推理时强制多视角像素级一致性,确保纹理无缝融合。该框架有效处理多种输入形式,结合先进神经架构与创新方法,产出高质量3D内容。报告详述系统架构、实验结果及未来改进方向,源代码与预训练权重已开源于https://github.com/Tencent/Tencent-XR-3DGen。
原文摘要 · Abstract (English)
This report presents a comprehensive framework for generating high-quality 3D shapes and textures from diverse input prompts, including single images, multi-view images, and text descriptions. The framework consists of 3D shape generation and texture generation. (1). The 3D shape generation pipeline employs a Variational Autoencoder (VAE) to encode implicit 3D geometries into a latent space and a diffusion network to generate latents conditioned on input prompts, with modifications to enhance model capacity. An alternative Artist-Created Mesh (AM) generation approach is also explored, yielding promising results for simpler geometries. (2). Texture generation involves a multi-stage process starting with frontal images generation followed by multi-view images generation, RGB-to-PBR texture conversion, and high-resolution multi-view texture refinement. A consistency scheduler is plugged into every stage, to enforce pixel-wise consistency among multi-view textures during inference, ensuring seamless integration. The pipeline demonstrates effective handling of diverse input formats, leveraging advanced neural architectures and novel methodologies to produce high-quality 3D content. This report details the system architecture, experimental results, and potential future directions to improve and expand the framework. The source code and pretrained weights are released at: https://github.com/Tencent/Tencent-XR-3DGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。