用实时变形噪声实现视频生成的精准运动控制
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
- 通过光流生成相关性噪声替代随机噪声，实现运动可控
- 可控制局部物体运动、全局镜头移动和动作迁移，效果自然
- 无需修改模型结构，适合快速集成到现有视频生成系统
生成模型旨在将随机噪声转化为结构化输出。本文通过结构化潜在噪声采样，增强视频扩散模型的运动控制能力。方法仅需对训练视频进行预处理以生成结构化噪声，不改变模型架构或训练流程。提出一种实时运行的新型噪声变形算法，将随机时间高斯性替换为由光流场导出的相关性变形噪声，同时保持空间高斯性。该算法高效，可低开销微调现代视频扩散基模型，提供一站式用户友好的运动控制方案：包括局部物体运动控制、全局相机运动控制和动作迁移。变形噪声在时间一致性与空间高斯性之间取得平衡，既实现有效运动控制，又保证单帧像素质量。大量实验与用户研究验证了方法优势，证明其在视频扩散模型运动控制中具有鲁棒性与可扩展性。视频结果见：https://eyeline-labs.github.io/Go-with-the-Flow。源码与模型检查点见：https://github.com/Eyeline-Labs/Go-with-the-Flow。
原文摘要 · Abstract (English)
Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise sampling. This is achieved by just a change in data: we pre-process training videos to yield structured noise. Consequently, our method is agnostic to diffusion model design, requiring no changes to model architectures or training pipelines. Specifically, we propose a novel noise warping algorithm, fast enough to run in real time, that replaces random temporal Gaussianity with correlated warped noise derived from optical flow fields, while preserving the spatial Gaussianity. The efficiency of our algorithm enables us to fine-tune modern video diffusion base models using warped noise with minimal overhead, and provide a one-stop solution for a wide range of user-friendly motion control: local object motion control, global camera movement control, and motion transfer. The harmonization between temporal coherence and spatial Gaussianity in our warped noise leads to effective motion control while maintaining per-frame pixel quality. Extensive experiments and user studies demonstrate the advantages of our method, making it a robust and scalable approach for controlling motion in video diffusion models. Video results are available on our webpage: https://eyeline-labs.github.io/Go-with-the-Flow. Source code and model checkpoints are available on GitHub: https://github.com/Eyeline-Labs/Go-with-the-Flow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。