用预训练视频扩散模型自动生成训练数据,实现精准视频编辑。
PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models
- 通过调节无分类器引导尺度,实时生成源帧与编辑帧的潜在表示对。
- 在多个视频编辑任务上超越现有方法,生成高质量、时序一致的动态变化。
- 适合需要高效训练且无成对数据的视频编辑研究者使用。
基于传播的视频编辑通过将单个编辑帧传播至后续帧,保持原有运动和结构信息,实现精细用户控制。然而,训练此类模型需大规模成对(源帧与编辑帧)视频数据集,获取成本高且复杂。为此,我们提出PropFly,一种基于预训练视频扩散模型(VDMs)实时生成监督信号的训练流程,无需依赖现成或预先计算的成对视频编辑数据集。具体地,利用不同无分类器引导(CFG)尺度下从中间加噪潜在表示中的一步去噪估计,实时合成多样化的‘源’(低CFG)与‘编辑’(高CFG)潜在表示对。源潜在表示作为视频结构信息,编辑潜在表示提供目标变换以供学习传播。该流程可附加适配器至预训练VDM,通过引导调制流匹配(GMFM)损失学习传播编辑,引导模型复现目标变换。实时监督确保模型学习到时空一致且动态的变换。大量实验表明,PropFly在多种视频编辑任务上显著优于当前最优方法,生成高质量编辑结果。
原文摘要 · Abstract (English)
Propagation-based video editing enables precise user control by propagating a single edited frame into following frames while maintaining the original context such as motion and structures. However, training such models requires large-scale, paired (source and edited) video datasets, which are costly and complex to acquire. Hence, we propose the PropFly, a training pipeline for Propagation-based video editing, relying on on-the-Fly supervision from pre-trained video diffusion models (VDMs) instead of requiring off-the-shelf or precomputed paired video editing datasets. Specifically, our PropFly leverages one-step clean latent estimations from intermediate noised latents with varying Classifier-Free Guidance (CFG) scales to synthesize diverse pairs of 'source' (low-CFG) and 'edited' (high-CFG) latents on-the-fly. The source latent serves as structural information of the video, while the edited latent provides the target transformation for learning propagation. Our pipeline enables an additional adapter attached to the pre-trained VDM to learn to propagate edits via Guidance-Modulated Flow Matching (GMFM) loss, which guides the model to replicate the target transformation. Our on-the-fly supervision ensures the model to learn temporally consistent and dynamic transformations. Extensive experiments demonstrate that our PropFly significantly outperforms the state-of-the-art methods on various video editing tasks, producing high-quality editing results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。