通过局部条件并行加速扩散模型推理,显著降低延迟。
Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference
- 仅依赖邻近区域部分信息进行并行计算
- 相比DistriFusion通信成本降低70%,速度提升2.36~8.02倍
- 适合对延迟敏感的高分辨率图像生成应用
扩散模型在图像生成和视频创作中表现出色,但其推理速度受限于缓慢的采样过程。生成单个样本需数十甚至上百次迭代,成为交互式应用和低延迟场景的主要瓶颈。为此,我们提出部分条件补丁并行(PCPP),以加速高分辨率扩散模型的推理。利用相邻扩散步骤间图像差异极小的特点,补丁并行(PP)通过多个GPU异步通信,基于前一步所有补丁的信息并行计算当前图像补丁。PCPP在此基础上改进,每步仅依赖邻近补丁的部分信息进行条件化,减少计算与通信开销。实验表明,相比最先进的PP实现DistriFusion,PCPP通信成本降低约70%,在4~8块GPU上实现2.36~8.02倍的推理加速,优于DistriFusion的2.32~6.71倍,代价是可能轻微降低图像质量。该方法展现了在高质量生成与低延迟之间取得良好权衡的潜力。
原文摘要 · Abstract (English)
Diffusion models have exhibited exciting capabilities in generating images and are also very promising for video creation. However, the inference speed of diffusion models is limited by the slow sampling process, restricting its use cases. The sequential denoising steps required for generating a single sample could take tens or hundreds of iterations and thus have become a significant bottleneck. This limitation is more salient for applications that are interactive in nature or require small latency. To address this challenge, we propose Partially Conditioned Patch Parallelism (PCPP) to accelerate the inference of high-resolution diffusion models. Using the fact that the difference between the images in adjacent diffusion steps is nearly zero, Patch Parallelism (PP) leverages multiple GPUs communicating asynchronously to compute patches of an image in multiple computing devices based on the entire image (all patches) in the previous diffusion step. PCPP develops PP to reduce computation in inference by conditioning only on parts of the neighboring patches in each diffusion step, which also decreases communication among computing devices. As a result, PCPP decreases the communication cost by around $70\%$ compared to DistriFusion (the state of the art implementation of PP) and achieves $2.36\sim 8.02\times$ inference speed-up using $4\sim 8$ GPUs compared to $2.32\sim 6.71\times$ achieved by DistriFusion depending on the computing device configuration and resolution of generation at the cost of a possible decrease in image quality. PCPP demonstrates the potential to strike a favorable trade-off, enabling high-quality image generation with substantially reduced latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。