用多光照扩散模型提升单图物体法向与材质估计精度
Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion
- 借助扩散模型生成不同方向的光照图像，降低估计不确定性
- 在合成重光照数据集上训练，实现高精度法向与PBR材质预测
- 适合需要真实感重光照的3D重建与材质编辑任务
从单张图像恢复物体几何与材质具有高度欠定性。本文提出Neural LightRig，通过利用2D扩散模型提供的光照先验，构建多光照扩散模型，在专用设计的合成重光照数据集上生成多个一致的光照图像，每幅图像由不同方向的点光源照明。利用这些多样化光照图像减少估计不确定性，采用U-Net主干的大型G-buffer模型进行训练，实现表面法向与材质的准确预测。大量实验表明，该方法显著优于现有最优方法，支持逼真的重光照效果。代码与数据集可在项目页面获取。
原文摘要 · Abstract (English)
Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature. In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors. Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs. This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions. 2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials. Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects. Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。