arXiv:2605.11869cs.CVcs.LG2026-05

不训练不修改模型,让视频生成提速2.4倍

FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity

论文配图:FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity
图 1 · 摘自论文原文
  • 通过帧间稀疏性策略,跳过部分帧计算但保持完整结构
  • 在万像2.2和混元视频1.5上实现2.11~2.41倍加速
  • 适合追求实时高清视频生成的开发者使用

尽管通过模型蒸馏可显著降低视频扩散变换器(DiT)的整体推理延迟,但每步推理延迟仍是关键瓶颈。现有加速方法主要利用去噪轨迹中的冗余,但在少步数场景下效果递减:因时间状态稀缺,难以有效复用特征或建模预测,形成加速障碍。为此,我们提出无需训练、与算子无关的帧间稀疏性扩散变换器(FIS-DiT),将优化焦点从时间轨迹转向潜在帧维度。该方法基于该维度的内在对偶性:帧级稀疏性允许减少计算,而每个帧位置对全局时空上下文同样重要。我们实现帧间稀疏性(FIS)作为执行策略,在模型层级操作帧子集,刷新所有潜在位置而不需全块计算。在万像2.2和混元视频1.5上的实证表明,FIS-DiT在VBench-Q和CLIP指标上几乎无损的情况下,持续实现2.11–2.41×加速,为实时高清视频生成提供可扩展且稳健的路径。

原文摘要 · Abstract (English)

While the overall inference latency of Video Diffusion Transformers (DiTs) can be substantially reduced through model distillation, per-step inference latency remains a critical bottleneck. Existing acceleration paradigms primarily exploit redundancy across the denoising trajectory; however, we identify a limitation where these step-wise strategies encounter diminishing returns in few-step regimes. In such scenarios, the scarcity of temporal states prevents effective feature reuse or predictive modeling, creating a formidable barrier to further acceleration. To overcome this, we propose Frame Interleaved Sparsity DiT (FIS-DiT), a training-free and operator-agnostic framework that shifts the optimization focus from the temporal trajectory to the latent frame dimension. Our approach is motivated by an intrinsic duality within this dimension: the existence of frame-wise sparsity that permits reduced computation, coupled with a structural consistency where each frame position remains equally vital to the global spatiotemporal context. Leveraging this insight, we implement Frame Interleaved Sparsity (FIS) as an execution strategy that manipulates frame subsets across the model hierarchy, refreshing all latent positions without requiring full-scale block computation. Empirical evaluations on Wan 2.2 and HunyuanVideo 1.5 demonstrate that FIS-DiT consistently achieves 2.11--2.41$\times$ speedup with negligible degradation across VBench-Q and CLIP metrics, providing a scalable and robust pathway toward real-time high-definition video generation.

视频生成扩散模型加速推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。