arXiv:2410.08151cs.CVcs.LG2024-10CVPR被引 73

提出渐进噪声调度,实现60秒高质量视频自回归生成

Progressive Autoregressive Video Diffusion Models

论文配图:Progressive Autoregressive Video Diffusion Models
图 1 · 摘自论文原文
  • 用逐帧递增噪声水平替代统一噪声,实现更平滑的注意力对齐
  • 在1440帧(60秒)视频生成中保持高画质,质量退化极小
  • 适合需要长时序一致性的视频生成研究者使用

当前前沿视频扩散模型虽能生成高质量视频,但受限于训练计算开销,通常只能生成约10秒(240帧)短片段。现有方法通过将前一段结尾直接作为下一段条件来实现自回归生成,导致场景突变、运动不自然及误差累积。本文重新审视视频扩散模型中的噪声水平假设,提出新思路:1. 为每帧分配逐帧递增的噪声水平,而非单一噪声;2. 以小步间隔逐步去噪并移位,而非一次性处理。该方法增强了相邻噪声水平帧间的注意力对应性,扩大了注意力窗口重叠,促进信息从早期帧向晚期帧有效传播。配备此渐进噪声调度的视频扩散模型可实现更高保真度的自回归长视频生成,且随时间质量衰减极小。本文首次展示文本条件下的60秒(1440帧)长视频生成结果,质量接近前沿模型。代码与视频结果见https://desaixie.github.io/pa-vdm/。

原文摘要 · Abstract (English)

Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos. However, they can only generate short video clips, normally around 10 seconds or 240 frames, due to computation limitations during training. Existing methods naively achieve autoregressive long video generation by directly placing the ending of the previous clip at the front of the attention window as conditioning, which leads to abrupt scene changes, unnatural motion, and error accumulation. In this work, we introduce a more natural formulation of autoregressive long video generation by revisiting the noise level assumption in video diffusion models. Our key idea is to 1. assign the frames with per-frame, progressively increasing noise levels rather than a single noise level and 2. denoise and shift the frames in small intervals rather than all at once. This allows for smoother attention correspondence among frames with adjacent noise levels, larger overlaps between the attention windows, and better propagation of information from the earlier to the later frames. Video diffusion models equipped with our progressive noise schedule can autoregressively generate long videos with much improved fidelity compared to the baselines and minimal quality degradation over time. We present the first results on text-conditioned 60-second (1440 frames) long video generation at a quality close to frontier models. Code and video results are available at https://desaixie.github.io/pa-vdm/.

视频生成扩散模型自回归长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。