用神经网络初始化物理渲染，实现单图材质编辑与真实光影效果。
Materialist: Physically Based Editing Using Single-Image Inverse Rendering
- 先用神经网络预测材质初始参数，再通过可微渲染逐步优化。
- 在合成与真实图像上均表现优异，对域外图像也具鲁棒性。
- 适合需要真实物理光照的图像编辑场景，如换材质、加物体。
实现物理一致的图像编辑仍是计算机视觉中的重大挑战。现有方法多依赖神经网络，难以准确处理阴影与折射；而基于物理的逆向渲染通常需多视角优化，不适用于单图场景。本文提出 Materialist，一种基于神经网络初始化的单图逆向渲染物理渲染流程。不同于以往用物理引导神经生成的方法，本方法利用神经网络预测初始材质属性，并通过渐进式可微渲染进行严格优化。该方法支持材质编辑、物体插入与重光照等多种应用，还提出一种无需完整场景几何信息即可通过光线追踪折射编辑透明度的有效方法。此外，其环境贴图估计方法也达到竞争性性能，进一步提升图像编辑精度。实验表明，该方法在合成与真实世界数据集上均表现强劲，甚至在挑战性的域外图像上也表现良好。
原文摘要 · Abstract (English)
Achieving physically consistent image editing remains a significant challenge in computer vision. Existing image editing methods typically rely on neural networks, which struggle to accurately handle shadows and refractions. Conversely, physics-based inverse rendering often requires multi-view optimization, limiting its practicality in single-image scenarios. In this paper, we propose Materialist, a neural-initialized physically based rendering pipeline for single-image inverse rendering. Unlike previous hybrid methods that use physics to guide neural generation, our method leverages neural networks to predict initial material properties, which are then rigorously optimized via progressive differentiable rendering. Our approach enables a range of applications, including material editing, object insertion, and relighting, while also introducing an effective method for editing material transparency via ray-traced refraction without requiring full scene geometry. Furthermore, our envmap estimation method also achieves competitive performance, further enhancing the accuracy of image editing task. Experiments demonstrate strong performance across synthetic and real-world datasets, excelling even on challenging out-of-domain images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。