arXiv:2511.06271cs.CV2025-11被引 8

用多平面光照图实现视频精准调光,保持内容一致且光影真实。

RelightMaster: Precise Video Relighting with Multi-plane Light Images

  • 提出MPLI光照提示,用多层平面表征3D光源位置强度与颜色。
  • 在自建数据集上实现光影变化时内容不变的高质量视频重照明。
  • 适配预训练扩散模型,无需重新训练即可实现可控光照编辑。

扩散模型虽推动了高质量视频生成与编辑,但精确且一致的视频重照明仍缺乏探索。主流文本到视频模型受限于文本对光照细节描述能力不足,且缺乏光照相关预训练。构建高质量重照明训练数据亦困难,因真实可控光照数据稀缺。为此,我们提出RelightMaster框架:首先,基于Unreal Engine构建首个在不同精确光照条件下保持动态内容一致的RelightVideo数据集;其次,引入多平面光照图(MPLI),受多平面图像启发,通过K个深度对齐平面建模3D光源的位置、强度与颜色,支持多光源场景并可泛化至未见光照设置;第三,设计光照图适配器,将MPLI通过预训练视频VAE压缩后注入视频扩散变换器(DiT)块中,利用基础模型生成先验,避免灾难性遗忘。实验表明,RelightMaster生成的光照与阴影具物理合理性,并有效保留原始场景内容。演示地址:https://wkbian.github.io/Projects/RelightMaster/

原文摘要 · Abstract (English)

Recent advances in diffusion models enable high-quality video generation and editing, but precise relighting with consistent video contents, which is critical for shaping scene atmosphere and viewer attention, remains unexplored. Mainstream text-to-video (T2V) models lack fine-grained lighting control due to text's inherent limitation in describing lighting details and insufficient pre-training on lighting-related prompts. Additionally, constructing high-quality relighting training data is challenging, as real-world controllable lighting data is scarce. To address these issues, we propose RelightMaster, a novel framework for accurate and controllable video relighting. First, we build RelightVideo, the first dataset with identical dynamic content under varying precise lighting conditions based on the Unreal Engine. Then, we introduce Multi-plane Light Image (MPLI), a novel visual prompt inspired by Multi-Plane Image (MPI). MPLI models lighting via K depth-aligned planes, representing 3D light source positions, intensities, and colors while supporting multi-source scenarios and generalizing to unseen light setups. Third, we design a Light Image Adapter that seamlessly injects MPLI into pre-trained Video Diffusion Transformers (DiT): it compresses MPLI via a pre-trained Video VAE and injects latent light features into DiT blocks, leveraging the base model's generative prior without catastrophic forgetting. Experiments show that RelightMaster generates physically plausible lighting and shadows and preserves original scene content. Demos are available at https://wkbian.github.io/Projects/RelightMaster/.

视频重照明扩散模型光照控制多平面表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。