用3D结构引导视频生成,让长视频更连贯。
WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion
- 用3D高斯点云动态维护场景结构,确保视角变化时几何一致。
- 引入时空扩散模型,对遮挡区域生成新内容,对已有区域精细优化。
- 适合需要高几何一致性长视频生成的研究者与开发者。
生成长时序、几何一致的视频面临根本矛盾:一致性要求严格遵循像素空间中的3D几何,而当前最先进的生成模型在相机条件化的潜在空间中表现最佳。这种脱节导致现有方法在遮挡区域和复杂摄像机轨迹下表现不佳。为此,我们提出WorldWarp框架,将3D结构锚点与2D生成修复器结合。通过高斯泼溅(3DGS)在线构建3D几何缓存,显式将历史内容投影至新视角,作为结构支架,确保每帧均遵循先前几何。然而静态投影不可避免留下空洞与伪影。我们采用时空扩散(ST-Diff)模型实现“填空与修正”目标,关键创新在于时空可变的噪声调度:空白区域接收全噪声以触发生成,已投影区域接收部分噪声以支持精细化调整。通过每一步动态更新3D缓存,WorldWarp保持跨视频片段的一致性。因此,在保证3D逻辑主导结构的同时,利用扩散逻辑完善纹理,实现了当前最优保真度。
原文摘要 · Abstract (English)
Generating long-range, geometrically consistent video presents a fundamental dilemma: while consistency demands strict adherence to 3D geometry in pixel space, state-of-the-art generative models operate most effectively in a camera-conditioned latent space. This disconnect causes current methods to struggle with occluded areas and complex camera trajectories. To bridge this gap, we propose WorldWarp, a framework that couples a 3D structural anchor with a 2D generative refiner. To establish geometric grounding, WorldWarp maintains an online 3D geometric cache built via Gaussian Splatting (3DGS). By explicitly warping historical content into novel views, this cache acts as a structural scaffold, ensuring each new frame respects prior geometry. However, static warping inevitably leaves holes and artifacts due to occlusions. We address this using a Spatio-Temporal Diffusion (ST-Diff) model designed for a "fill-and-revise" objective. Our key innovation is a spatio-temporal varying noise schedule: blank regions receive full noise to trigger generation, while warped regions receive partial noise to enable refinement. By dynamically updating the 3D cache at every step, WorldWarp maintains consistency across video chunks. Consequently, it achieves state-of-the-art fidelity by ensuring that 3D logic guides structure while diffusion logic perfects texture. Project page: \href{https://hyokong.github.io/worldwarp-page/}{https://hyokong.github.io/worldwarp-page/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。