无需训练即可生成更长视频,直接拼接预训练模型输出。
VideoMerge: Towards Training-free Long Video Generation
- 用拼接方式合并短视频生成结果,不需重新训练模型。
- 支持任意长度视频生成,保持时序一致性和内容连贯性。
- 适合需要快速扩展视频长度的研究者与创作者。
长视频生成是计算机视觉中的挑战性课题。基于扩散模型的生成方法虽在质量上达到领先水平,但其训练成本高昂,且通常固定输入视频的空间与时间维度。尽管已有多个开源预训练视频扩散模型,但它们在推理时指定的视频长度若未在训练中覆盖,便难以生成高质量长视频。此外,长视频因时长增加导致内容复杂度上升,易出现时序不一致问题。本文提出 VideoMerge,一种无需训练的方法,可无缝融合由预训练文本到视频扩散模型生成的短片段。该方法在不损失原始模型表达力的前提下,支持用户自定义视频长度和动态变化内容。通过协同运用多种正交策略,有效提升生成视频的流畅性、一致性与动态表现力。
原文摘要 · Abstract (English)
Long video generation remains a challenging and compelling topic in computer vision. Diffusion based models, among the various approaches to video generation, have achieved state of the art quality with their iterative denoising procedures. However, the intrinsic complexity of the video domain renders the training of such diffusion models exceedingly expensive in terms of both data curation and computational resources. Moreover, these models typically operate on a fixed noise tensor that represents the video, resulting in predetermined spatial and temporal dimensions. Although several high quality open-source pretrained video diffusion models, jointly trained on images and videos of varying lengths and resolutions, are available, it is generally not recommended to specify a video length at inference that was not included in the training set. Consequently, these models are not readily adaptable to the direct generation of longer videos by merely increasing the specified video length. In addition to feasibility challenges, long-video generation also encounters quality issues. The domain of long videos is inherently more complex than that of short videos: extended durations introduce greater variability and necessitate long-range temporal consistency, thereby increasing the overall difficulty of the task. We propose VideoMerge, a training-free method that can be seamlessly adapted to merge short videos generated by pretrained text-to-video diffusion model. Our approach preserves the model's original expressiveness and consistency while allowing for extended duration and dynamic variation as specified by the user. By leveraging the strengths of pretrained models, our method addresses challenges related to smoothness, consistency, and dynamic content through orthogonal strategies that operate collaboratively to achieve superior quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。