arXiv:2509.13848cs.CVcs.LG2025-09AAAI被引 2

通过自推测机制提升扩散模型推理速度,兼顾精度与效率。

SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation

  • 利用自推测信息与历史特征动态选择缓存内容
  • 在Stable Diffusion 3/3.5/FLUX上实现2.8到3.17倍加速
  • 无需训练,适合追求高效推理的部署场景

特征缓存近期成为加速扩散模型推理的有前景方法,能有效缓解高计算需求带来的效率瓶颈。本文从信息利用角度分析现有方法,指出仅依赖历史信息会限制精度与速度表现。为此提出新范式:基于不同迭代步中同时间步的信息相似性引入未来信息,实现自推测。在此基础上,提出无需训练的多级特征缓存策略SpecDiff,包含基于自推测信息的特征选择算法和基于特征重要性得分的多级分类算法。实验表明,在NVIDIA A800-80GB GPU上,SpecDiff相较RFlow在Stable Diffusion 3、3.5和FLUX上分别实现平均2.80×、2.74×和3.17×加速,且质量损失可忽略。通过融合推测与历史信息,突破速度与精度权衡瓶颈,推动高效扩散模型推理的帕累托前沿。

原文摘要 · Abstract (English)

Feature caching has recently emerged as a promising method for diffusion model acceleration. It effectively alleviates the inefficiency problem caused by high computational requirements by caching similar features in the inference process of the diffusion model. In this paper, we analyze existing feature caching methods from the perspective of information utilization, and point out that relying solely on historical information will lead to constrained accuracy and speed performance. And we propose a novel paradigm that introduces future information via self-speculation based on the information similarity at the same time step across different iteration times. Based on this paradigm, we present \textit{SpecDiff}, a training-free multi-level feature caching strategy including a cached feature selection algorithm and a multi-level feature classification algorithm. (1) Feature selection algorithm based on self-speculative information. \textit{SpecDiff} determines a dynamic importance score for each token based on self-speculative information and historical information, and performs cached feature selection through the importance score. (2) Multi-level feature classification algorithm based on feature importance scores. \textit{SpecDiff} classifies tokens by leveraging the differences in feature importance scores and introduces a multi-level feature calculation strategy. Extensive experiments show that \textit{SpecDiff} achieves average 2.80 \times, 2.74 \times , and 3.17\times speedup with negligible quality loss in Stable Diffusion 3, 3.5, and FLUX compared to RFlow on NVIDIA A800-80GB GPU. By merging speculative and historical information, \textit{SpecDiff} overcomes the speedup-accuracy trade-off bottleneck, pushing the Pareto frontier of speedup and accuracy in the efficient diffusion model inference.

扩散模型推理加速特征缓存自推测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。