从单图生成可控相机、物体和天气的视频,实现世界状态统一控制。
Holo-World: Unified Camera, Object and Weather Control for Video World Model

- 基于单张图像联合控制相机、物体与天气,实现场景结构保持。
- 在多个天气条件下生成视频,保持物体运动一致性且结构稳定。
- 适合需要精准控制环境变化的视频生成与虚拟世界构建任务。
视频世界模型正朝着在可控制相机与物体运动下维持观测世界的同时,允许环境状态变化的方向发展。然而,这些控制仍相互隔离,且天气生成通常依赖已有源视频或重建场景。本文研究一种首帧锚定的源到状态设定:模型从单张图像出发,遵循显式的相机与物体控制及可选天气指令,生成视频以保留源世界或迁移到目标天气状态。为此,我们构建了HoloStateData数据集,将多样化视频转化为统一的相机、物体与天气控制样本。进一步提出Holo-World模型,从单图联合控制场景。其统一场景适配器将世界保持与天气迁移分解为独立参数子空间,利用渲染背景、几何缓冲区与物体控制保持结构,同时建模天气相关的外观与粒子效果。此外,场景-天气解耦的CFG机制分别引导场景与天气残差,强化目标天气效果而不过度放大整体条件。定量与定性实验表明,Holo-World在精确控制相机与物体运动的同时,保持一致的场景结构,并成功迁移到多种目标天气状态,在天气状态生成上优于视频到视频的天气编辑基线。
原文摘要 · Abstract (English)
Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://xiangchenyin.github.io/Holo-World/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。