分块生成长视频,解决显存不足问题
Towards Chunk-Wise Generation for Long Videos
- 将长视频拆成短片段逐块生成,降低显存占用
- 提出k步搜索策略,提升分块间时序连贯性
- 适合需要生成超长视频的研究者与开发者
长视频生成因时空域复杂性和巨大张量计算带来的显存需求而面临挑战。基于扩散模型的生成方法虽在视频生成任务中表现优异,但通常以预设分辨率和长度训练,推理时需一次性处理整个视频张量,易引发显存溢出(OOM)。一种解决方案是采用自回归方式逐块生成短视频片段,并保持块间强时空关联,再拼接成完整长视频。本文系统调研了基于自回归分块策略的长视频生成方法,分析了将短时图像到视频模型直接用于长视频任务时的常见问题,并设计了一种高效的k步搜索方案以缓解这些问题。
原文摘要 · Abstract (English)
Generating long-duration videos has always been a significant challenge due to the inherent complexity of spatio-temporal domain and the substantial GPU memory demands required to calculate huge size tensors. While diffusion based generative models achieve state-of-the-art performance in video generation task, they are typically trained with predefined video resolutions and lengths. During inference, a noise tensor with specific resolution and length should be specified at first, and the model will perform denoising on the entire video tensor simultaneously, all the frames together. Such approach will easily raise an out-of-memory (OOM) problem when the specified resolution and/or length exceed a certain limit. One of the solutions to this problem is to generate many short video chunks autoregressively with strong inter-chunk spatio-temporal relation and then concatenate them together to form a long video. In this approach, a long video generation task is divided into multiple short video generation subtasks, and the cost of each subtask is reduced to a feasible level. In this paper, we conduct a detailed survey on long video generation with the autoregressive chunk-by-chunk strategy. We address common problems caused by applying short image-to-video models to long video tasks and design an efficient $k$-step search solution to mitigate these problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。