用卷积优化让扩散模型在各类设备上更快生成图像
Open-Source Acceleration of Stable-Diffusion.cpp Deployable on All Devices
- 引入快速卷积算法提升2D卷积效率，突破推理瓶颈
- 在M1 Pro上实现单层加速2.76倍，整体生成提速4.79倍
- 兼容主流模型如SDXL、SDXL-Turbo，适合本地部署需求
Stable Diffusion 在生成高质量图像中起关键作用，但其图像生成过程耗时且内存占用高。为解决此问题，stable-diffusion.cpp（Sdcpp）作为高效推理框架应运而生。尽管轻量，当前 Sdcpp 中 ggml_conv_2d 算子性能不佳，导致推理延迟高、内存消耗大。本文提出基于 Winograd 算法的优化版本，显著加速2D卷积操作——这是流程中的主要瓶颈。通过分析依赖与独立计算图，充分利用设备局部性和并行性，实现性能大幅提升。所提框架在多种 Stable Diffusion 模型（包括 SDv1.4、v1.5、v2.1、SDXL 及 SDXL-Turbo）上均输出正确端到端结果。评估表明，在 M1 Pro 上，单个卷积层最高加速 2.76 倍，整体图像生成速度最高提升 4.79 倍。
原文摘要 · Abstract (English)
Stable diffusion plays a crucial role in generating high-quality images. However, image generation is time-consuming and memory-intensive. To address this, stable-diffusion.cpp (Sdcpp) emerges as an efficient inference framework to accelerate the diffusion models. Although it is lightweight, the current implementation of ggml_conv_2d operator in Sdcpp is suboptimal, exhibiting both high inference latency and massive memory usage. To address this, in this work, we present an optimized version of Sdcpp leveraging the Winograd algorithm to accelerate 2D convolution operations, which is the primary bottleneck in the pipeline. By analyzing both dependent and independent computation graphs, we exploit the device's locality and parallelism to achieve substantial performance improvements. Our framework delivers correct end-to-end results across various stable diffusion models, including SDv1.4, v1.5, v2.1, SDXL, and SDXL-Turbo. Our evaluation results demonstrate a speedup up to 2.76x for individual convolutional layers and an inference speedup up to 4.79x for the overall image generation process, compared with the original Sdcpp on M1 pro. Homepage: https://github.com/SealAILab/stable-diffusion-cpp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。