用重心外推加速扩散Transformer，提速超3倍且不占显存
Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache
- 用重心外推法预测每步去噪结果，避免重复计算
- 相比基线模型最高提速3.30倍，显存占用几乎不变
- 无需训练，适合部署在资源受限的推理场景
扩散Transformer在图像和视频生成中表现优异，但其迭代采样过程计算开销大，每步去噪都需要大量矩阵运算。现有基于缓存的加速方法虽能减少冗余计算，却因存储中间状态而增加显存占用，直接限制了推理批大小。本文提出一种无需训练的加速方法，通过重心外推器对扩散Transformer采样进行分步预测。利用重心外推，预测器在数值上稳定，有效缓解前向预测中的振荡伪影（类似龙格现象）。在图像和视频生成任务上的广泛实验表明，该方法在显存使用与感知质量之间取得良好平衡，相较于基线扩散Transformer推理，端到端采样速度最高提升3.30倍。
原文摘要 · Abstract (English)
Diffusion Transformers achieve high-fidelity image and video generation, but their iterative sampling remains expensive, for each denoising step requires large matrix operations. Existing cache-based acceleration reduces redundant computation yet increases the VRAM footprint by storing intermediate states, which can directly constrain inference batch size. In this work, we propose a training-free acceleration method that performs stepwise forecasting for DiT sampling using a Barycentric Extrapolator. By leveraging barycentric extrapolation, our predictor is numerically stable and alleviates oscillatory artifacts analogous to the Runge phenomenon during forward forecasting. Across extensive experiments on both image and video generation, our approach provides a favorable trade-off between memory usage and perceptual quality, while delivering up to 3.30x end-to-end sampling speedup compared with baseline DiT inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。