用文字生成图像实现人体前后双视角纹理,无需真实配对照片。
SMPL-GPTexture: Dual-View 3D Human Texture Estimation using Text-to-Image Generation Models
- 输入文字提示生成人体前后视角图像,解决真实配对图像难获取问题。
- 通过逆向投影将图像颜色映射到3D模型UV空间,生成完整高精度纹理。
- 结合扩散修复与融合机制,自动补全缺失区域,适合虚拟人建模应用。
为3D虚拟人生成高质量、逼真的纹理仍是计算机视觉与多媒体领域的核心挑战。然而,因隐私、伦理及采集成本问题,真实的人体前后配对图像极为稀缺,限制了数据可扩展性。此外,利用深度生成模型(如GAN或扩散模型)从单张图像学习先验以推断未见区域(如人体背面)时,常导致伪影、结构不一致或细节丢失。为此,我们提出SMPL-GPTexture(基于皮肤多人物线性模型的通用纹理生成),一种新方法:输入自然语言提示,借助先进文本到图像生成模型,生成高分辨率人体前后视角图像作为纹理估计起点。首先,使用人体网格恢复模型建立图像像素与3D模型UV坐标的鲁棒2D到3D对齐。其次,采用逆向光栅化技术,显式将输入图像中的颜色投影至UV空间,生成精确完整的纹理图。最后,通过扩散模型进行修复,融合机制整合结果,生成统一的全幅纹理图。大量实验表明,SMPL-GPTexture能生成与用户提示高度匹配的高分辨率纹理。
原文摘要 · Abstract (English)
Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。