通过动态重用特征加速扩散模型推理，速度翻倍且质量几乎不变。
Sortblock: Similarity-Aware Feature Reuse for Diffusion Model
- 根据相邻步骤的残差相似度动态决定是否跳过计算。
- 在多个DiT架构上实现2倍以上加速，图像质量损失极小。
- 无需重新训练，适合实时生成场景的部署需求。
扩散Transformer（DiTs）在生成任务中展现出卓越能力，尤其受益于Transformer结构带来的视觉与艺术保真度提升。然而，其固有的逐步去噪过程导致推理延迟高，限制了在实时场景中的应用。现有免训练加速方法通常在固定时间步或层重复使用中间特征，忽略了去噪阶段语义关注点的动态变化。为此，我们提出Sortblock，一种免训练的推理加速框架，通过基于相邻时间步间特征相似度动态缓存块级特征。该方法通过排序残差演化过程，自适应确定重计算比例，在保留生成质量的前提下选择性跳过冗余计算。此外，引入轻量级线性预测机制以减少跳过块累积误差。在多种任务和DiT架构上的大量实验表明，Sortblock实现了超过2×的推理加速，同时输出质量下降可忽略，为扩散生成模型提供了高效且通用的加速方案。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks.To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks.Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2$\times$ inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。