用物理渲染生成视频代理,实现可控且连贯的视频重布光。
LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

- 先用PBR渲染生成带目标光照的视频代理,再通过扩散模型翻译成最终结果。
- 在真实世界基准上超越现有最佳方法,合成数据提升全局光照建模能力。
- 适合需要精细光照控制和长视频一致性的影视/游戏场景重布光应用。
视频重布光需平衡长时间一致性与物理光照传输的准确性,依赖对材质、几何和光照等场景属性的精确估计。现有方法分为两类:(1) 通过逆向渲染重建视频光度特性,再用正向渲染转换至目标光照,使用物理渲染(PBR)或神经渲染器,但存在重建噪声问题,难以建模全局光照等复杂效果;(2) 将任务视为以目标环境图或文本为条件的生成式视频到视频翻译,受限于扩散模型处理长视频的能力不足,且依赖成对训练数据。我们提出LightCrafter,一种混合管道,将视频重布光重构为对代理视频的翻译:不直接转换输入视频,而是将输入在目标光照下的PBR渲染结果翻译至最终目标。该方法将光照目标嵌入代理视频中,无需让扩散模型学习环境图等光照概念,从而实现更精细的光照控制,并自然保证长时间一致性。我们发现仅使用PBR渲染已优于部分现有方法,但在全局光照等效果上仍有不足;为此,通过后训练方式在合成视频对和真实未配对视频上微调CogVideoX,引入光度先验。我们在现有真实世界重布光基准上超越当前最优,同时贡献一个合成基准用于进一步分析。代码、数据集、评估指标将公开。
原文摘要 · Abstract (English)
Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。