让单目视频同时控制视角和光照,生成更真实动态画面。
Light-X: Generative 4D Video Rendering with Camera and Illumination Control
- 分离几何与光照信号,用动态点云和重光照帧分别建模。
- 合成多视角多光照数据对,训练出能联合控制视角与光照的模型。
- 适合需要灵活操控视频视觉效果的研究者和创作者。
近期光照控制进展将基于图像的方法扩展至视频领域,但仍面临光照保真度与时间一致性之间的权衡。要实现真实场景的生成建模,关键在于同时控制相机轨迹与光照,因为视觉动态本质上由几何结构和光照共同决定。为此,我们提出 Light-X,一种可从单目视频中实现视角与光照联合控制的视频生成框架。1)提出解耦设计,将几何与光照信号分离:通过用户定义的相机轨迹投影动态点云捕捉几何与运动,同时将一致重光照帧投影至相同几何结构以提供光照线索。这种显式、细粒度的引导机制有效实现解耦并提升光照质量。2)为解决缺乏成对多视角与多光照视频的问题,引入 Light-Syn,一种基于退化与逆映射的合成管道,从真实世界单目视频中生成训练数据对。该策略构建的数据集涵盖静态、动态及生成场景,确保训练鲁棒性。大量实验表明,Light-X 在联合相机-光照控制上优于基线方法,并在文本与背景条件设置下超越先前视频重光照方法。
原文摘要 · Abstract (English)
Recent advances in illumination control extend image-based methods to video, yet still facing a trade-off between lighting fidelity and temporal consistency. Moving beyond relighting, a key step toward generative modeling of real-world scenes is the joint control of camera trajectory and illumination, since visual dynamics are inherently shaped by both geometry and lighting. To this end, we present Light-X, a video generation framework that enables controllable rendering from monocular videos with both viewpoint and illumination control. 1) We propose a disentangled design that decouples geometry and lighting signals: geometry and motion are captured via dynamic point clouds projected along user-defined camera trajectories, while illumination cues are provided by a relit frame consistently projected into the same geometry. These explicit, fine-grained cues enable effective disentanglement and guide high-quality illumination. 2) To address the lack of paired multi-view and multi-illumination videos, we introduce Light-Syn, a degradation-based pipeline with inverse-mapping that synthesizes training pairs from in-the-wild monocular footage. This strategy yields a dataset covering static, dynamic, and AI-generated scenes, ensuring robust training. Extensive experiments show that Light-X outperforms baseline methods in joint camera-illumination control and surpasses prior video relighting methods under both text- and background-conditioned settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。