用光流引导提示优化，让视频生成更连贯自然。
Optical-Flow Guided Prompt Optimization for Coherent Video Generation
- 通过训练判别器区分真实与生成视频的光流，指导提示优化。
- 在反向采样中优化可学习提示嵌入，提升视频时序一致性。
- 适用于多种扩散模型，适合追求高质量连贯视频的研究者。
尽管文本到视频的扩散模型已取得显著进展，但仍面临生成视频时序不一致的问题。在扩散框架中，引导技术已被证明能有效提升推理阶段的输出质量；然而，将其应用于视频扩散模型时，需处理整个序列的计算复杂性。为此，我们提出一种名为MotionPrompt的新框架，通过光流引导视频生成过程。具体而言，我们训练一个判别器以区分真实视频与生成视频中随机帧对的光流。由于提示会影响整个视频，我们在反向采样步骤中利用训练好的判别器对随机帧对的梯度，优化可学习的提示嵌入。该方法能够生成视觉上连贯、贴近自然运动动态的视频序列，同时不降低生成内容的保真度。我们在多种模型上验证了该方法的有效性。
原文摘要 · Abstract (English)
While text-to-video diffusion models have made significant strides, many still face challenges in generating videos with temporal consistency. Within diffusion frameworks, guidance techniques have proven effective in enhancing output quality during inference; however, applying these methods to video diffusion models introduces additional complexity of handling computations across entire sequences. To address this, we propose a novel framework called MotionPrompt that guides the video generation process via optical flow. Specifically, we train a discriminator to distinguish optical flow between random pairs of frames from real videos and generated ones. Given that prompts can influence the entire video, we optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs. This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content. We demonstrate the effectiveness of our approach across various models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。