arXiv:2503.17539cs.CV2025-03ICCV被引 1

提出视频接口网络,让长视频生成更快更稳。

Generating, Fast and Slow: Scalable Parallel Video Generation with Video Interface Networks

  • 用接口网络并行处理视频块,全局语义引导局部去噪
  • 生成长视频时计算量减少25%-40%,运动更平滑
  • 适合需要高效、连贯长视频生成的场景

扩散Transformer(DiTs)可生成短时逼真视频,但直接对长视频进行全注意力训练与采样仍存在计算挑战。现有方法将长视频分段生成,需多次采样迭代和一致性模块。为此,我们提出视频接口网络(VINs),在DiTs基础上引入抽象模块,实现视频块的并行推理。每个扩散步骤中,VIN从局部块的噪声输入中编码全局语义,并以编码表示引导DiTs并行去噪。VIN与DiT通过去噪目标端到端联合学习。此外,VIN采用固定大小的编码标记,通过单次交叉注意力编码输入,解耦编码与输入,从而支持长视频生成并学习关键语义。VBench实验表明,VIN在背景一致性与主体连贯性上优于现有分块方法。光学流分析显示,本方法在使用25%-40%更少浮点运算的情况下达到当前最优运动平滑性。用户研究中,人类评估者对视频整体质量与时间一致性给予积极评价。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) can generate short photorealistic videos, yet directly training and sampling longer videos with full attention across the video remains computationally challenging. Alternative methods break long videos down into sequential generation of short video segments, requiring multiple sampling chain iterations and specialized consistency modules. To overcome these challenges, we introduce a new paradigm called Video Interface Networks (VINs), which augment DiTs with an abstraction module to enable parallel inference of video chunks. At each diffusion step, VINs encode global semantics from the noisy input of local chunks and the encoded representations, in turn, guide DiTs in denoising chunks in parallel. The coupling of VIN and DiT is learned end-to-end on the denoising objective. Further, the VIN architecture maintains fixed-size encoding tokens that encode the input via a single cross-attention step. Disentangling the encoding tokens from the input thus enables VIN to scale to long videos and learn essential semantics. Experiments on VBench demonstrate that VINs surpass existing chunk-based methods in preserving background consistency and subject coherence. We then show via an optical flow analysis that our approach attains state-of-the-art motion smoothness while using 25-40% fewer FLOPs than full generation. Finally, human raters favorably assessed the overall video quality and temporal consistency of our method in a user study.

视频生成扩散模型并行推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。