用统一框架从任意位置和时间补全视频,无需重训练模型。
VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning
- 通过混合条件策略分离时空控制,保持输入分布一致
- 在多个任务上达到当前最佳性能,涵盖场景内保真与跨场景创意
- 适用于需要灵活视频生成的创作者与研究者
现有可控视频生成方法通常针对特定任务设计,如首帧图像生成视频、修复或插值,将时空控制视为孤立问题。本文提出统一任务——任意时空视频补全,即模型可根据用户指定的任意空间位置和时间戳的片段生成连贯视频。然而,在现代潜空间视频扩散模型中实现此目标极具挑战:因果视频变分自编码器(VAE)将多帧压缩为单一潜在槽,导致帧级条件设定本质病态;直接输入稀疏填充、零值填充的视频会导致严重分布外伪影。为此,我们提出VideoCanvas,一种无需修改或重新训练VAE的简单而高效框架,采用上下文条件范式实现任意时空补全。核心思想是混合条件策略:空间上以图像模式编码零填充全帧画布,确保VAE输入分布一致;时间上使用时间RoPE插值,为每个条件分配连续分数索引,实现精确帧级对齐。为评估该能力,我们构建了VideoCanvasBench,首个面向任意时空视频补全的基准,涵盖场景内保真度与跨场景创造力。大量实验表明,VideoCanvas在单一统一框架下,在多样化视频生成任务中均达当前最优表现。
原文摘要 · Abstract (English)
Existing controllable video generation methods are typically designed for rigid, task-specific settings, such as first-frame image-to-video, inpainting, or interpolation, treating spatio-temporal control as a set of isolated problems. We formalize a unified task, arbitrary spatio-temporal video completion, where a model generates a coherent video from user-specified patches placed at any spatial location and timestamp. However, realizing such a unified framework within modern latent video diffusion models is non-trivial: causal video VAEs compress multiple frames into a single latent slot, making frame-level conditioning fundamentally ill-posed, and directly feeding sparsely populated, zero-padded video inputs into the VAE leads to severe out-of-distribution artifacts. To address these challenges, we propose VideoCanvas, a simple yet effective framework that adapts the In-Context Conditioning paradigm to arbitrary spatio-temporal completion without modifying or retraining the VAE. Our key idea is a hybrid conditioning strategy that decouples spatial and temporal control: spatially, we encode zero-padded full-frame canvases in image mode to keep VAE inputs in-distribution, and temporally we use Temporal RoPE Interpolation to assign each condition a continuous fractional index in the latent sequence for precise frame-level alignment. To evaluate this capability, we develop VideoCanvasBench, the first benchmark for arbitrary spatio-temporal video completion, covering both intra-scene fidelity and inter-scene creativity. Extensive experiments demonstrate that VideoCanvas achieves state-of-the-art performance across a diverse range of video generation tasks under a single, unified framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。