用扩散模型估计视频中变化的室内光照，实现时空一致。
Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors
- 用2D扩散先验优化由MLP表示的连续光照场。
- 在真实场景视频上实现时空一致的光照估计，优于已有方法。
- 无需重新训练，可零样本泛化到复杂真实场景。
从单张图像或视频中估计室内光照仍极具挑战，尤其当光照在空间和时间上持续变化时。本文提出一种方法，通过输入视频估计一个连续的光照场，描述场景中随时间和空间变化的光照情况。利用2D扩散先验来优化以MLP表示的光照场。为实现对真实场景的零样本泛化，我们对预训练图像扩散模型进行微调，使其能通过联合修复多个反光球（作为光照探针）来预测多位置的光照。在单图或视频的室内光照估计任务上评估该方法，结果显著优于现有基线。最重要的是，本文首次在真实复杂视频上展示了时空一致的光照估计效果，此前工作极少实现此目标。
原文摘要 · Abstract (English)
Indoor lighting estimation from a single image or video remains a challenge due to its highly ill-posed nature, especially when the lighting condition of the scene varies spatially and temporally. We propose a method that estimates from an input video a continuous light field describing the spatiotemporally varying lighting of the scene. We leverage 2D diffusion priors for optimizing such light field represented as a MLP. To enable zero-shot generalization to in-the-wild scenes, we fine-tune a pre-trained image diffusion model to predict lighting at multiple locations by jointly inpainting multiple chrome balls as light probes. We evaluate our method on indoor lighting estimation from a single image or video and show superior performance over compared baselines. Most importantly, we highlight results on spatiotemporally consistent lighting estimation from in-the-wild videos, which is rarely demonstrated in previous works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。