将视频生成中的因果推理与去噪过程分离,提升效率且不损失质量。
Causality in Video Diffusers is Separable from Denoising
- 用因果编码器处理帧间依赖,轻量解码器逐帧去噪。
- 实验显示速度更快,每帧延迟降低30%以上,生成质量相当。
- 适合需要高效视频生成的工业场景或实时应用。
因果性——指组件间的时序单向因果关系——是视频、语言和机器人轨迹等复杂生成过程的基础。当前的因果扩散模型将时序推理与多步去噪纠缠在一起,在所有层、每个去噪步骤和整个上下文中均使用因果注意力。本文通过系统探测自回归视频扩散模型,发现两个关键规律:(1)早期层在不同去噪步骤中产生高度相似的特征,表明扩散轨迹上存在冗余计算;(2)深层网络表现出稀疏的跨帧注意力,主要完成帧内渲染。基于此,我们提出可分离因果扩散(SCD),其显式分离每帧一次的时序推理(通过因果变压器编码器)与多步帧级渲染(通过轻量扩散解码器)。在合成与真实基准上的预训练和微调任务中,大量实验表明,SCD显著提升了吞吐量和每帧延迟,同时生成质量达到或超过强基线模型。
原文摘要 · Abstract (English)
Causality -- referring to temporal, uni-directional cause-effect relationships between components -- underlies many complex generative processes, including videos, language, and robot trajectories. Current causal diffusion models entangle temporal reasoning with iterative denoising, applying causal attention across all layers, at every denoising step, and over the entire context. In this paper, we show that the causal reasoning in these models is separable from the multi-step denoising process. Through systematic probing of autoregressive video diffusers, we uncover two key regularities: (1) early layers produce highly similar features across denoising steps, indicating redundant computation along the diffusion trajectory; and (2) deeper layers exhibit sparse cross-frame attention and primarily perform intra-frame rendering. Motivated by these findings, we introduce Separable Causal Diffusion (SCD), a new architecture that explicitly decouples once-per-frame temporal reasoning, via a causal transformer encoder, from multi-step frame-wise rendering, via a lightweight diffusion decoder. Extensive experiments on both pretraining and post-training tasks across synthetic and real benchmarks show that SCD significantly improves throughput and per-frame latency while matching or surpassing the generation quality of strong causal diffusion baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。