利用相邻步骤的输出一致性,实现无需训练的扩散采样加速。
Accelerating Diffusion Sampling via Exploiting Local Transition Coherence
- 发现相邻采样步骤间输出的统计相干性,用于估计转移算子。
- 在稳定扩散模型上提速1.67倍,视频生成提速1.55倍。
- 兼容主流加速方法,适合需要实时生成的场景。
基于文本的扩散模型在从文本描述生成高质量图像和视频方面取得了显著进展。然而,去噪过程的漫长采样时间仍是实际应用中的主要瓶颈。以往方法要么忽略相邻步骤间的统计关系,要么依赖注意力或特征相似性,且通常仅适用于特定网络结构。为此,我们发现相邻步骤间转移算子中存在一种新的统计关系,聚焦于网络输出之间的关联,该关系不依赖网络结构。基于此,提出无需训练的加速方法LTC-Accel,利用该关系通过相邻步骤估计当前转移算子。由于不依赖特定网络结构,LTC-Accel可适配几乎所有基于扩散的方法,且与现有加速技术正交,易于集成。实验表明,LTC-Accel在文生图和文生视频任务中显著提升采样速度,同时保持良好生成质量。具体而言,在Stable Diffusion v2上实现1.67倍加速,在视频生成模型上实现1.55倍加速;与蒸馏模型结合时,视频生成可达10倍加速,支持超过16FPS的实时生成。
原文摘要 · Abstract (English)
Text-based diffusion models have made significant breakthroughs in generating high-quality images and videos from textual descriptions. However, the lengthy sampling time of the denoising process remains a significant bottleneck in practical applications. Previous methods either ignore the statistical relationships between adjacent steps or rely on attention or feature similarity between them, which often only works with specific network structures. To address this issue, we discover a new statistical relationship in the transition operator between adjacent steps, focusing on the relationship of the outputs from the network. This relationship does not impose any requirements on the network structure. Based on this observation, we propose a novel training-free acceleration method called LTC-Accel, which uses the identified relationship to estimate the current transition operator based on adjacent steps. Due to no specific assumptions regarding the network structure, LTC-Accel is applicable to almost all diffusion-based methods and orthogonal to almost all existing acceleration techniques, making it easy to combine with them. Experimental results demonstrate that LTC-Accel significantly speeds up sampling in text-to-image and text-to-video synthesis while maintaining competitive sample quality. Specifically, LTC-Accel achieves a speedup of 1.67-fold in Stable Diffusion v2 and a speedup of 1.55-fold in video generation models. When combined with distillation models, LTC-Accel achieves a remarkable 10-fold speedup in video generation, allowing real-time generation of more than 16FPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。