单图重建高保真可重光照3D形象,用扩散模型分步生成材质贴图
Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading

- 分阶段用扩散模型的先验,在UV空间逐步补全纹理、去光照、分解材质
- 仅用不到100个真实3D扫描训练,生成4K级逼真PBR贴图
- 通过可微分着色损失保证物理合理性,适合影视与游戏角色建模
从一张野外拍摄的单图重建高保真、可重光照的3D形象是一个困难的病态问题,主要受限于高质量PBR数据稀缺以及光照与固有材质的解耦复杂性。本文提出一种数据高效框架,利用统一预训练扩散模型的强大先验,分步解决纹理补全、去光照和材质分解。不同于依赖碎片化流程或大量专有数据的方法,我们采用级联低秩适配(LoRAs)在UV空间中为每个子任务适配扩散模型。首先使用图像修复LoRA补全遮挡导致的缺失UV纹理,借助模型语义理解生成语义与光度一致的细节。随后引入光照均一化LoRA与新型跨固有注意力机制,协同合成像素对齐的PBR贴图(反照率、法线、粗糙度、镜面反射、位移)。为确保物理合理性,分解阶段引入基于UV空间的可微分BRDF着色损失,迫使生成过程遵循渲染方程,避免传统栅格化监督带来的伪影。大量实验表明,本方法在少于100个真实3D扫描上训练,即可生成分辨率高达4K的完整且逼真的PBR资产,性能优于现有最优方法,所有训练代码与模型权重将在论文接受后公开。
原文摘要 · Abstract (English)
Reconstructing high-fidelity, relightable 3D avatars from a single in-the-wild image is a challenging ill-posed problem, primarily hindered by the scarcity of high-quality PBR data and the complexity of disentangling illumination from intrinsic materials. In this paper, we present a data-efficient framework that leverages the robust priors of a unified pre-trained diffusion backbone to sequentially address texture completion, delighting, and material decomposition. Unlike existing methods that rely on fragmented pipelines or extensive proprietary datasets, we utilize cascaded Low-Rank Adaptations (LoRAs) to adapt the strong generative prior of the diffusion model for each sub-task in UV space. Specifically, we first employ an Inpainting LoRA to complete missing UV textures caused by occlusion, leveraging the model's semantic understanding to generate semantically and photometrically coherent details. Subsequently, a Light-Homogenization LoRA and a novel Cross-Intrinsic Attention mechanism are introduced to remove baked-in lighting and collaboratively synthesize pixel-aligned PBR maps (Albedo, Normal, Roughness, Specular, and Displacement). To ensure physical plausibility, we impose a UV-space differentiable BRDF shading loss during the decomposition stage, forcing the generative process to adhere to the rendering equation without the artifacts typical of rasterization-based supervision. Extensive experiments demonstrate that our method, trained on fewer than 100 real 3D scans, generates comprehensive, 4K-resolution PBR assets with superior realism and generalization compared to state-of-the-art methods, and all training code and model weights will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。