让照片在真实场景下实现物理可控的光照重渲染。
Physically Controllable Relighting of Photographs
- 用单目图像估计几何与材质，构建带颜色的三维网格
- 通过路径追踪生成初步光照结果，再由神经网络优化成写实效果
- 无需标注数据，自监督训练实现真实场景下的光照自由调整
我们提出一种自监督方法，实现真实场景照片的物理可控制光照重渲染。通过单目图像估计几何和固有属性，重建场景的彩色网格表示，用户可在3D中自由定义光照配置。新光照下的场景先经路径追踪引擎渲染，再输入前馈神经渲染器生成最终写实效果。我们设计了可微分渲染流程，从原始图像集合中重建真实光照，实现神经渲染器的自监督训练。该方法将传统3D图形工具（如Blender）中的显式光照控制能力，首次引入真实拍摄场景的光照编辑中。
原文摘要 · Abstract (English)
We present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing. We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering. Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components. This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine. We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result. We develop a differentiable rendering process to reconstruct in-the-wild scene illumination, enabling self-supervised training of our neural renderer on raw image collections. Our method represents a significant step in bringing the explicit physical control over lights available in typical 3D computer graphics tools, such as Blender, to in-the-wild relighting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。