无需训练即可加速视频扩散模型，保持甚至提升画质。
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration
- 通过非对称删减冗余特征令牌，实现无损加速。
- 在多个数据集上提速最高达4.2倍，画质下降小于0.5%。
- 适用于各类视频扩散模型，特别适合资源受限场景。
扩散变换器（DiTs）在生成高质量视频方面表现优异，但计算开销巨大。现有加速方法常依赖昂贵的微调或泛化能力有限。本文提出无需训练、模型无关的非对称删减与恢复（AsymRnR）方法。该方法基于观察：DiTs中不同层、去噪步骤和特征类型存在显著的特征令牌冗余差异。AsymRnR在注意力计算中不对称地减少冗余令牌，实现加速且输出质量几乎不变，部分情况下甚至有所提升。我们还设计了自适应的删减调度策略，并引入匹配缓存以进一步加速删减过程。理论分析与大量实验验证表明，AsymRnR可无缝集成至当前最优视频DiTs，带来显著提速。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have proven effective in generating high-quality videos but are hindered by high computational costs. Existing video DiT sampling acceleration methods often rely on costly fine-tuning or exhibit limited generalization capabilities. We propose Asymmetric Reduction and Restoration (AsymRnR), a training-free and model-agnostic method to accelerate video DiTs. It builds on the observation that redundancies of feature tokens in DiTs vary significantly across different model blocks, denoising steps, and feature types. Our AsymRnR asymmetrically reduces redundant tokens in the attention operation, achieving acceleration with negligible degradation in output quality and, in some cases, even improving it. We also tailored a reduction schedule to distribute the reduction across components adaptively. To further accelerate this process, we introduce a matching cache for more efficient reduction. Backed by theoretical foundations and extensive experimental validation, AsymRnR integrates into state-of-the-art video DiTs and offers substantial speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。