用记忆机制连接短视频片段,生成上千帧的连贯长视频。
Inference-based GAN Video Generation
- 构建变分自编码器与GAN融合的视频生成框架
- 通过马尔可夫链实现数百至数千帧的连续视频生成
- 适合需要长时序动态视频的应用场景
视频生成在生成式深度学习推动下取得显著进展,但长序列生成仍是重大挑战。生成视频需具备连贯、连续且有意义的运动。现有GAN、VAE和扩散模型多用于生成16帧以内的短视频。本文提出一种新型无条件视频生成模型,结合对抗训练与变分编码器,形成类似VAE-GAN的混合结构。该模型包含内容与运动两个处理分支,但传统方法在视频长度扩展时质量下降明显。为此,本文提出一种新的内存高效方法,通过马尔可夫链框架与回溯机制,将每个状态定义为一个短时视频生成器,实现视频子序列的顺序连接,有效保持时间依赖性,生成具有时序一致性与动态性的长视频序列。
原文摘要 · Abstract (English)
Video generation has seen remarkable progress thanks to advancements in generative deep learning. However, generating long sequences remains a significant challenge. Generated videos should not only display coherent and continuous movement but also meaningful movement in successions of scenes. Models such as GANs, VAEs, and Diffusion Networks have been used for generating short video sequences, typically up to 16 frames. In this paper, we first propose a new type of video generator by enabling adversarial-based unconditional video generators with a variational encoder, akin to a VAE-GAN hybrid structure. The proposed model, as in other video deep learning-based processing frameworks, incorporates two processing branches, one for content and another for movement. However, existing models struggle with the temporal scaling of the generated videos. Classical approaches often result in degraded video quality when attempting to increase the generated video length, especially for significantly long sequences. To overcome this limitation, our research study extends the initially proposed VAE-GAN video generation model by employing a novel, memory-efficient approach to generate long videos composed of hundreds or thousands of frames ensuring their temporal continuity, consistency and dynamics. Our approach leverages a Markov chain framework with a recall mechanism, where each state represents a short-length VAE-GAN video generator. This setup enables the sequential connection of generated video sub-sequences, maintaining temporal dependencies and resulting in meaningful long video sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。