用扩散Transformer实现视频外扩,兼顾时空一致性。
OutDreamer: Video Outpainting with a Diffusion Transformer
- 采用双分支结构,分离控制与生成,提升外扩精度。
- 零样本测试下超越现有方法,在多个基准上表现最优。
- 适合需要高质量视频扩展的应用,如影视制作、虚拟场景生成。
视频外扩是一项挑战性任务,需在原视频边界外生成连贯的新内容,要求兼具时空一致性。现有主流方法多基于带有U-Net主干的潜在扩散模型,但仍难以在生成质量与适应性上取得突破。扩散Transformer(DiT)因其优异性能成为有前景的替代方案。本文提出基于DiT的视频外扩框架OutDreamer,包含两个核心组件:高效视频控制分支用于提取被遮掩视频信息,条件外扩分支则基于提取的条件生成缺失内容。此外,提出一种掩码驱动自注意力层,动态融合掩码信息,增强模型对不同外扩任务的适应性。进一步引入潜在对齐损失,以维持帧内及帧间整体一致性。针对长视频外扩,设计跨视频片段重构器,迭代生成缺失内容,保障视频片段间的时序一致性。大量实验表明,所提零样本OutDreamer在广泛认可的基准上优于现有先进零样本方法。
原文摘要 · Abstract (English)
Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many state-of-the-art methods utilize latent diffusion models with U-Net backbones but still struggle to achieve high quality and adaptability in generated content. Diffusion transformers (DiTs) have emerged as a promising alternative because of their superior performance. We introduce OutDreamer, a DiT-based video outpainting framework comprising two main components: an efficient video control branch and a conditional outpainting branch. The efficient video control branch effectively extracts masked video information, while the conditional outpainting branch generates missing content based on these extracted conditions. Additionally, we propose a mask-driven self-attention layer that dynamically integrates the given mask information, further enhancing the model's adaptability to outpainting tasks. Furthermore, we introduce a latent alignment loss to maintain overall consistency both within and between frames. For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content, ensuring temporal consistency across video clips. Extensive evaluations demonstrate that our zero-shot OutDreamer outperforms state-of-the-art zero-shot methods on widely recognized benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。