通过稳定性准则自适应加速扩散模型采样,速度提升超1.8倍且质量几乎无损。
SADA: Stability-guided Adaptive Diffusion Acceleration
- 基于采样轨迹和微分方程求解原理,统一决策每步与每令牌的稀疏性。
- 在SD-2、SDXL、Flux上实现≥1.8倍加速,LPIPS≤0.10,FID≤4.5。
- 无需修改即可适配ControlNet、MusicLDM等多模态任务,适合高效生成场景。
扩散模型在生成任务中表现卓越,但因迭代采样和二次注意力计算导致高耗时。现有免训练加速方法虽降低每步开销,却显著牺牲生成质量。本文提出稳定性引导的自适应扩散加速(SADA),通过单一稳定性准则统一步级与令牌级稀疏决策,加速基于常微分方程的生成模型(如扩散与流匹配)。针对不同提示词导致的去噪轨迹差异,SADA动态调整稀疏策略;同时利用数值微分方程求解器的精确梯度信息,设计合理近似方案。在SD-2、SDXL、Flux上使用EDM与DPM++求解器的全面评估显示,相较原始模型实现≥1.8倍加速,仅带来极小质量损失(LPIPS ≤ 0.10,FID ≤ 4.5),显著优于已有方法。此外,SADA可无缝适配其他生成管线:无需修改即加速ControlNet,并使MusicLDM提速1.8倍,频谱LPIPS仅增加约0.01。
原文摘要 · Abstract (English)
Diffusion models have achieved remarkable success in generative tasks but suffer from high computational costs due to their iterative sampling process and quadratic attention costs. Existing training-free acceleration strategies that reduce per-step computation cost, while effectively reducing sampling time, demonstrate low faithfulness compared to the original baseline. We hypothesize that this fidelity gap arises because (a) different prompts correspond to varying denoising trajectory, and (b) such methods do not consider the underlying ODE formulation and its numerical solution. In this paper, we propose Stability-guided Adaptive Diffusion Acceleration (SADA), a novel paradigm that unifies step-wise and token-wise sparsity decisions via a single stability criterion to accelerate sampling of ODE-based generative models (Diffusion and Flow-matching). For (a), SADA adaptively allocates sparsity based on the sampling trajectory. For (b), SADA introduces principled approximation schemes that leverage the precise gradient information from the numerical ODE solver. Comprehensive evaluations on SD-2, SDXL, and Flux using both EDM and DPM++ solvers reveal consistent $\ge 1.8\times$ speedups with minimal fidelity degradation (LPIPS $\leq 0.10$ and FID $\leq 4.5$) compared to unmodified baselines, significantly outperforming prior methods. Moreover, SADA adapts seamlessly to other pipelines and modalities: It accelerates ControlNet without any modifications and speeds up MusicLDM by $1.8\times$ with $\sim 0.01$ spectrogram LPIPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。