无需微调即可提升文本到视频生成的准确性，通过规划草图引导生成。
Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
- 用多模态规划生成视频草图，预设物体位置与运动轨迹。
- 通过噪声反演与去噪引导，使生成视频更贴合文本描述。
- 无需额外计算资源，适合大模型部署，适配多种生成框架。
近期文本到视频（T2V）扩散模型在视觉质量上取得显著进展，但仍难以准确遵循复杂文本指令，尤其在空间布局或物体运动轨迹控制方面。现有方法需在推理时微调或迭代修改注意力图，大幅增加内存开销，限制大模型应用。为此，我们提出 Video-MSG，一种无需训练的 T2V 引导方法，基于多模态规划与结构化噪声初始化。该方法分三步：前两步生成视频草图——即精细时空规划的初稿视频帧，明确背景、前景及物体轨迹；第三步通过噪声反演与去噪，以视频草图引导下游 T2V 扩散模型生成最终视频。Video-MSG 不需要推理时微调或注意力操作，避免额外内存消耗，便于集成大模型。在 T2VCompBench 与 VBench 等主流基准上，对 VideoCrafter2 与 CogVideoX-5B 多种骨干模型验证了其有效性。并通过详尽消融实验分析噪声反演比例、背景生成器、背景物体检测与前景分割的影响。
原文摘要 · Abstract (English)
Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。