让视频生成提前“看未来”，避免相机穿墙崩溃。
Generative View Stitching
- 并行采样整个视频序列，用未来视角引导当前生成。
- 在预设相机路径下实现无碰撞、帧间一致的稳定生成。
- 兼容任意现成扩散模型，适合影视动画与虚拟拍摄场景。
自回归视频扩散模型虽能生成长序列且保持历史一致性，却无法利用未来信息进行当前生成，导致在预设相机轨迹下的视频生成中出现与场景碰撞，进而迅速崩溃。为此，我们提出生成视图拼接（GVS），通过并行采样整个序列，使生成场景忠实于预设相机轨迹的每个部分。核心贡献是一种扩展自机器人规划中扩散拼接方法的采样算法。不同于通常需专门训练模型的方法，GVS可兼容任意基于扩散强制（Diffusion Forcing）训练的现成视频模型，该框架已具备拼接所需的特性。我们进一步提出全时引导（Omni Guidance），通过同时结合过去和未来信息增强时间一致性，并支持环路闭合机制以实现长程连贯性。GVS在多种预设相机路径下均表现稳定、无碰撞、帧间一致，且能闭环生成，包括奥斯卡·雷托尔瓦德的不可能楼梯。结果详见：https://andrewsonga.github.io/gvs。
原文摘要 · Abstract (English)
Autoregressive video diffusion models are capable of long rollouts that are stable and consistent with history, but they are unable to guide the current generation with conditioning from the future. In camera-guided video generation with a predefined camera trajectory, this limitation leads to collisions with the generated scene, after which autoregression quickly collapses. To address this, we propose Generative View Stitching (GVS), which samples the entire sequence in parallel such that the generated scene is faithful to every part of the predefined camera trajectory. Our main contribution is a sampling algorithm that extends prior work on diffusion stitching for robot planning to video generation. While such stitching methods usually require a specially trained model, GVS is compatible with any off-the-shelf video model trained with Diffusion Forcing, a prevalent sequence diffusion framework that we show already provides the affordances necessary for stitching. We then introduce Omni Guidance, a technique that enhances the temporal consistency in stitching by conditioning on both the past and future, and that enables our proposed loop-closing mechanism for delivering long-range coherence. Overall, GVS achieves camera-guided video generation that is stable, collision-free, frame-to-frame consistent, and closes loops for a variety of predefined camera paths, including Oscar Reutersvärd's Impossible Staircase. Results are best viewed as videos at https://andrewsonga.github.io/gvs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。