根据运动速度差异分配不同去噪步数，显著提升视频生成效率
Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
- 按时空令牌运动速度动态分配去噪步数，低速区域减少计算
- 在25%运行时长下仍保持高质量，优于现有缓存方法
- 无需离线调优，适合资源受限的实时视频生成场景
扩散Transformer（DiT）虽实现了顶尖视频生成质量，但因对序列中每个时空令牌统一应用相同去噪步数而带来巨大计算开销。人类视觉会忽略大量冗余运动，为何模型仍同等对待所有令牌？本文提出无需训练的推理算法HSA，根据令牌的速度动态分配不同步数预算。为解决由此产生的序列长度不匹配问题，HSA引入KV缓存同步机制，使活跃令牌可访问完整序列，同时完全跳过静止令牌。此外，我们推导出一种缓存欧拉更新，可在单次操作中推进被跳过的令牌潜在状态，无需额外模型评估。我们在Wan-2和LTX-2模型上评估了HSA在文本到视频（T2V）与图像到视频（I2V）生成任务中的表现。结果表明，HSA显著优于先前最优缓存方法及原始流匹配基线，尤其在激进加速场景（如50%和25%运行时间）下优势明显。关键的是，HSA在不依赖昂贵离线分析的情况下，实现了更优的质量-运行时间权衡，在严苛计算预算下依然有效保持结构完整性和生成质量。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have achieved state-of-the-art video generation quality, but they incur immense computational cost because standard inference applies the same number of denoising steps uniformly to every token in the sequence. It is well known that human vision ignores vast amounts of redundant motion. Why, then, do our densest models treat every spatiotemporal token with equal priority? In this paper, we introduce Heterogeneous Step Allocation (HSA), a training-free inference algorithm that assigns varying step budgets to different spatiotemporal tokens based on their velocity dynamics. To resolve the resulting sequence-length mismatch without sacrificing global context, HSA introduces a KV-cache synchronization mechanism that allows active tokens to attend to the full sequence while entirely bypassing inactive tokens. Furthermore, we derive a cached Euler update that advances the latent states of skipped tokens in a single operation without additional model evaluations. We evaluate HSA on the Wan-2 and LTX-2 models for both text-to-video (T2V) and image-to-video (I2V) generation. Our results demonstrate that HSA significantly outperforms previous state-of-the-art caching methods and the vanilla Flow Matching baseline, especially at aggressive acceleration regimes (e.g., 50% and 25% runtimes). Crucially, HSA achieves a superior quality-runtime Pareto frontier without the need for expensive offline profiling, robustly preserving structural integrity and generation quality even under tight computational budgets. Project page: https://ernestchu.github.io/hsa
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。