分离光照与物体辐射,实现动态光影下的场景编辑。
RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing
- 用预录的稳定光照场景作为参考,解耦动态光照影响。
- 通过可学习光照向量,精准分离物体自身辐射与环境光色。
- 支持交互式掩码和光流正则化,适合影视特效与虚拟拍摄应用。
尽管神经辐射场取得显著进展,动态光照变化问题仍未解决。现有方法常将时空域中的时间相关/无关成分参数化,但物体辐射与自身发光及光照颜色高度耦合。本文提出RehearsalNeRF,一种在剧烈光照变化下学习解耦神经场的新方法。核心思想是利用预先采集的稳定光照场景(如排练舞台)作为参考,强制不同光照条件下的几何一致性。RehearsalNeRF采用可学习的光照向量,在时间维度上表征光照颜色,用于将投影光色从场景辐射中解耦。此外,该方法仅需使用现成的交互式掩码即可重建动态物体的神经场。为解耦动态物体,我们提出一种基于光流的新型正则化,提供粗粒度监督以辅助颜色解耦。实验表明,RehearsalNeRF在动态光照条件下的新视角合成与场景编辑任务中均表现出强鲁棒性。源代码与视频数据集将公开发布。
原文摘要 · Abstract (English)
Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。