arXiv:2506.04213cs.CV2025-06被引 18

提升视频生成效率,让复杂控制更快速

FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers

  • 动态筛选关键条件标记,缩短处理序列
  • 减少条件与视频的重复计算,提速2-3倍
  • 适用于视频生成与编辑,效果更好或相当

细粒度且高效的可控性在视频扩散模型中日益受到关注。最近,上下文条件化成为统一条件视频生成的强大范式,通过将不同条件信号与噪声视频潜在表示拼接成长序列,并通过全注意力机制联合处理,如FullDiT。尽管有效,这些方法随任务复杂度增加面临二次方计算开销,阻碍实际部署。本文系统分析原上下文条件化框架中的效率瓶颈,识别出两个关键原因:条件标记内部的固有冗余,以及扩散过程中条件-潜码交互的计算冗余。基于此,提出FullDiT2,一种面向通用可控性的高效上下文条件化框架,从两方面创新:首先,采用动态标记选择机制自适应识别重要条件标记,减少统一全注意力的序列长度;其次,设计选择性条件缓存机制,最小化条件标记与视频潜码间的冗余交互。在六个不同条件视频生成与编辑任务上的大量实验表明,FullDiT2实现显著计算量降低,每扩散步平均耗时减少2-3倍,视频生成质量无明显下降甚至有所提升。

原文摘要 · Abstract (English)

Fine-grained and efficient controllability on video diffusion transformers has raised increasing desires for the applicability. Recently, In-context Conditioning emerged as a powerful paradigm for unified conditional video generation, which enables diverse controls by concatenating varying context conditioning signals with noisy video latents into a long unified token sequence and jointly processing them via full-attention, e.g., FullDiT. Despite their effectiveness, these methods face quadratic computation overhead as task complexity increases, hindering practical deployment. In this paper, we study the efficiency bottleneck neglected in original in-context conditioning video generation framework. We begin with systematic analysis to identify two key sources of the computation inefficiencies: the inherent redundancy within context condition tokens and the computational redundancy in context-latent interactions throughout the diffusion process. Based on these insights, we propose FullDiT2, an efficient in-context conditioning framework for general controllability in both video generation and editing tasks, which innovates from two key perspectives. Firstly, to address the token redundancy, FullDiT2 leverages a dynamic token selection mechanism to adaptively identify important context tokens, reducing the sequence length for unified full-attention. Additionally, a selective context caching mechanism is devised to minimize redundant interactions between condition tokens and video latents. Extensive experiments on six diverse conditional video editing and generation tasks demonstrate that FullDiT2 achieves significant computation reduction and 2-3 times speedup in averaged time cost per diffusion step, with minimal degradation or even higher performance in video generation quality. The project page is at \href{https://fulldit2.github.io/}{https://fulldit2.github.io/}.

视频生成扩散模型效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。