用预训练文生图模型生成视频，保持创意多样性并提升时序一致性。
HARIVO: Harnessing Text-to-Image Models for Video Generation
- 基于冻结文生图模型，设计帧级令牌与映射网络架构。
- 引入时序平滑损失与梯度采样优化，生成更连贯的视频。
- 兼容ControlNet、DreamBooth等工具，适合快速部署视频生成。
我们提出一种方法，从预训练的文生图（T2I）模型构建基于扩散的视频生成模型。受AnimateDiff启发，该方法冻结原有T2I模型，仅训练时间层。我们进一步设计了专用架构，包含映射网络和帧级令牌，以适配视频生成，并保留原始T2I模型的多样性和创造性。关键创新包括用于时序平滑的新损失函数及梯度采样缓解技术，有效应对公开视频数据有限的问题，确保生成视频的现实性与时序一致性。通过在架构与损失函数中融入视频特有的归纳偏置，我们的方法基于冻结的StableDiffusion模型，简化训练流程，并可无缝集成ControlNet、DreamBooth等现成工具。
原文摘要 · Abstract (English)
We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: https://kwonminki.github.io/HARIVO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。