arXiv:2608.01771cs.CV2026-08中稿 · TOG 2026被引 1

实时视频重光照,3D灯光可交互控制。

LiveLight: Real-time Streaming Video Relighting with Interactive Control

论文配图:LiveLight: Real-time Streaming Video Relighting with Interactive Control
图 1 · 摘自论文原文
  • 用轻量适配器注入3D光照信息,实现精准动态调光。
  • 低NFE下仍保高质量,支持实时运行。
  • 支持连续流式交互,适合影视与VR应用。

我们提出LiveLight,首个基于扩散模型的实时流式视频重光照框架,支持交互式3D灯光控制。该任务面临三大挑战:如何有效将动态3D光照注入扩散模型、在极低NFE(函数求值次数)预算下保持高保真生成、以及实现连续流式交互。为此,我们设计三个关键组件:第一,提出轻量适配器,直接将多平面光照辐照(MPLI)条件——即编码3D光照几何的深度感知辐照图——输入扩散主干;第二,引入几何引导反馈分支,利用冻结的几何估计器在训练阶段强制深度与法线一致性,确保阴影几何合理且无推理开销;第三,设计渐进滚动窗口策略,维护不同噪声水平的潜在块去噪阶梯,通过传播中间状态保证时间一致性,并支持任意长视频重光照及每帧参考刷新。在真实世界与合成基准上的实验表明,LiveLight在实时速度下达到顶尖重光照质量,显著优于离线基线,在时间稳定性、灯光可控性和用户偏好上表现更优。为推动实时交互重光照研究,我们将公开模型、训练数据及合成数据生成器。

原文摘要 · Abstract (English)

We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.

视频重光照扩散模型实时交互3D光照

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。