让光影重演更真实,用物理反馈提升图像视频的光照效果。
UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback
- 在视觉空间中引入深度与法向图反馈,确保光照符合场景结构。
- 相比传统方法,光照一致性显著提升,图像视频重光速率达20倍。
- 设计六维光照标注协议,支持细粒度控制与自动评估。
重光是兼具实用需求与艺术价值的重要任务,近期扩散模型在生成丰富可控的光照效果方面展现出巨大潜力。然而,这些模型通常在语义潜空间中优化,其邻近性不保证视觉空间中的物理真实性,常导致高光过曝、阴影错位、遮挡错误等问题。为此,我们提出UniLumos,一个统一处理图像与视频重光的框架,将RGB空间的几何反馈引入流匹配主干网络。通过从输出中提取深度图与法向图进行监督,显式对齐光照与场景结构,增强物理合理性。但该反馈需高质量输出作为监督信号,标准多步去噪计算成本高。为此,我们采用路径一致性学习,使监督在少步训练下仍有效。为实现精细控制与监督,我们设计了结构化的六维光照标注协议,捕获核心光照属性。基于此,提出LumosBench,一个解耦属性级别的基准,利用大规模视觉语言模型自动评估各维度的重光精度,实现可解释性评估。大量实验表明,UniLumos在重光质量上达到当前最优,物理一致性显著提升,同时在图像和视频重光上分别实现20倍加速。代码已开源:https://github.com/alibaba-damo-academy/Lumos-Custom。
原文摘要 · Abstract (English)
Relighting is a crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results, such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with UniLumos, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into a flow matching backbone. By supervising the model with depth and normal maps extracted from its outputs, we explicitly align lighting effects with the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective even under few-step training regimes. To enable fine-grained relighting control and supervision, we design a structured six-dimensional annotation protocol capturing core illumination attributes. Building upon this, we propose LumosBench, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision-language models, enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the-art relighting quality with significantly improved physical consistency, while delivering a 20x speedup for both image and video relighting. Code is available at https://github.com/alibaba-damo-academy/Lumos-Custom.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。