用多项式预测扩散模型特征,实现高速生成且画质更优
Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration
- 将扩散过程特征建模为时间函数,用切比雪夫多项式拟合预测
- 在FLUX.1上达4.79倍加速,万像2.1-14B达4.67倍,画质优于基线
- 无需训练,误差不随步长累积,适合高阶加速场景
扩散模型已成为高质量图像与视频生成的主流工具,但其推理速度受扩散变压器多次迭代的严重制约。为减少计算量,近期方法采用特征缓存重用策略,在选定步骤跳过网络计算,利用先前步骤的缓存特征。然而,此类方法仅依赖局部近似,导致大步跳过时误差迅速累积,显著降低生成质量。本文提出无训练的谱特征预测器(Spectrum),实现全局、长程特征重用并严格控制误差。具体地,将去噪器的潜在特征视为时间函数,用切比雪夫多项式进行逼近;通过岭回归拟合各基函数系数,进而预测多个未来扩散步骤的特征。理论上证明该方法具备更优的长时行为,且误差界不随步长累积。在多种先进图像与视频扩散模型上的实验一致验证其优越性:在FLUX.1上实现最高4.79倍加速,在万像2.1-14B上达4.67倍加速,同时保持显著更高的样本质量。
原文摘要 · Abstract (English)
Diffusion models have become the dominant tool for high-fidelity image and video generation, yet are critically bottlenecked by their inference speed due to the numerous iterative passes of Diffusion Transformers. To reduce the exhaustive compute, recent works resort to the feature caching and reusing scheme that skips network evaluations at selected diffusion steps by using cached features in previous steps. However, their preliminary design solely relies on local approximation, causing errors to grow rapidly with large skips and leading to degraded sample quality at high speedups. In this work, we propose spectral diffusion feature forecaster (Spectrum), a training-free approach that enables global, long-range feature reuse with tightly controlled error. In particular, we view the latent features of the denoiser as functions over time and approximate them with Chebyshev polynomials. Specifically, we fit the coefficient for each basis via ridge regression, which is then leveraged to forecast features at multiple future diffusion steps. We theoretically reveal that our approach admits more favorable long-horizon behavior and yields an error bound that does not compound with the step size. Extensive experiments on various state-of-the-art image and video diffusion models consistently verify the superiority of our approach. Notably, we achieve up to 4.79$\times$ speedup on FLUX.1 and 4.67$\times$ speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。