从一张随意拍摄的图片生成可重复使用的物理渲染材质
MatE: Material Extraction from Single-Image via Geometric Prior
- 利用深度图作为几何先验，分两步校正图像畸变
- 通过双分支扩散模型恢复材质的反照率、法线、粗糙度和高度图
- 对光照和视角不敏感，适合普通用户快速生成真实材质
高保真物理渲染（PBR）材质的创建仍是图形流水线中的瓶颈，通常需要专业设备和人工后期处理。为实现平民化，我们提出MatE，一种从单张真实场景图像中生成可重复使用PBR材质的新方法。给定一张图像和用户提供的掩码，MatE首先利用估计的深度图作为几何先验进行粗略校正，再通过一个双分支扩散模型，基于旋转对齐和尺度对齐的训练数据学习到的一致性，进一步修正残余畸变，并输出完整的材质贴图集，包括反照率、法线、粗糙度和高度图。该框架对输入图像的未知光照和视角具有不变性，能够从随意拍摄的照片中恢复内在材质属性。在合成与真实数据上的全面实验表明，该方法有效且鲁棒，使用户可基于真实图像轻松创建逼真材质。
原文摘要 · Abstract (English)
The creation of high-fidelity, physically-based rendering (PBR) materials remains a bottleneck in many graphics pipelines, typically requiring specialized equipment and expert-driven post-processing. To democratize this process, we present MatE, a novel method for generating tileable PBR materials from a single image taken under unconstrained, real-world conditions. Given an image and a user-provided mask, MatE first performs coarse rectification using an estimated depth map as a geometric prior, and then employs a dual-branch diffusion model. Leveraging a learned consistency from rotation-aligned and scale-aligned training data, this model further rectify residual distortions from the coarse result and translate it into a complete set of material maps, including albedo, normal, roughness and height. Our framework achieves invariance to the unknown illumination and perspective of the input image, allowing for the recovery of intrinsic material properties from casual captures. Through comprehensive experiments on both synthetic and real-world data, we demonstrate the efficacy and robustness of our approach, enabling users to create realistic materials from real-world image.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。