用分层隐变量实现长视频生成的一致性,节省计算量。
MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation

- 通过多尺度隐变量从粗到细生成视频帧
- 在自定义长Minecraft视频上显著提升一致性
- 适合需要长时序一致性的视频生成研究者
视频生成模型日益强大,但长序列一致性仍难实现,因数十帧即需过长的Transformer序列。本文提出在多尺度令牌空间中采用粗到细生成策略缓解此问题。首先预训练一个自动编码器,将每帧压缩为从典型潜空间到每帧仅数个令牌的层级隐变量:最粗层级捕捉场景布局与语义等关键信息,精细层级补充外观与纹理细节。随后训练视频扩散模型,按粗到细方式生成这些隐变量。通过控制每一步生成与上下文使用中的细节层次,有效保持几何结构与物体恒常性的一致性,同时减少对不显著细节的长程计算开销。在自定义的长Minecraft视频数据集上验证,该方法生成的视频滚动生成结果明显优于现有基线。
原文摘要 · Abstract (English)
Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information, such as scene layout and semantics, while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。