用视频扩散模型实现从视频中逆向推导光照与材质，再正向生成逼真图像。
DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
- 基于视频扩散模型，从真实视频中估计G-buffers
- 无需光路模拟即可生成高质量渲染图像
- 支持单视频输入下的重打光、材质编辑等应用
理解与建模光照效应是计算机视觉与图形学中的基础任务。传统物理渲染（PBR）虽能精确模拟光传输，但依赖精确的场景表示——如显式3D几何、高质量材质属性和光照条件，这些在真实场景中往往难以获取。为此，我们提出DiffusionRenderer，一种基于神经网络的统一框架，同时解决逆向与正向渲染问题。利用强大的视频扩散模型先验，逆向渲染模型可从真实视频中准确估计G-buffers，为图像编辑任务提供接口并生成训练数据；而正向渲染模型则无需显式光路模拟，即可从G-buffers生成逼真图像。实验表明，DiffusionRenderer能有效逼近逆向与正向渲染，持续优于当前最先进方法。该模型仅需单个视频输入，即可实现重打光、材质编辑与真实物体插入等实用应用。
原文摘要 · Abstract (English)
Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。