通过条件引导调度实现混合并行,加速扩散模型生成且保持画质。
Accelerating Diffusion via Hybrid Data-Pipeline Parallelism Based on Conditional Guidance Scheduling
- 基于条件与无条件去噪路径划分数据,创新性地实现并行
- 在两块3090上使SDXL提速2.31倍、SD3提速2.07倍
- 适用于多种架构,适合高分辨率生成场景
扩散模型在图像、视频和音频生成中取得显著进展,但推理仍计算昂贵。现有基于分布式并行的加速方法常引入明显生成伪影,且加速效果与GPU数量不匹配。为此,我们提出一种混合并行框架,结合新型条件数据划分策略与自适应流水线调度机制——自适应并行切换,以降低条件扩散模型的生成延迟并保持高质量。核心思想是:(i) 将条件与无条件去噪路径作为新的数据划分视角;(ii) 根据两条路径间的去噪差异动态启用最优流水线并行。该框架在使用两块NVIDIA RTX 3090 GPU时,对SDXL和SD3分别实现2.31倍和2.07倍的延迟降低,同时保持图像质量。结果表明该方法在基于U-Net的扩散模型和基于DiT的流匹配架构中均具通用性。其在高分辨率合成场景下的加速表现也优于现有方法。代码已开源:https://github.com/kaist-dmlab/Hybridiff。
原文摘要 · Abstract (English)
Diffusion models have achieved remarkable progress in high-fidelity image, video, and audio generation, yet inference remains computationally expensive. Nevertheless, current diffusion acceleration methods based on distributed parallelism suffer from noticeable generation artifacts and fail to achieve substantial acceleration proportional to the number of GPUs. Therefore, we propose a hybrid parallelism framework that combines a novel data parallel strategy, condition-based partitioning, with an optimal pipeline scheduling method, adaptive parallelism switching, to reduce generation latency and achieve high generation quality in conditional diffusion models. The key ideas are to (i) leverage the conditional and unconditional denoising paths as a new data-partitioning perspective and (ii) adaptively enable optimal pipeline parallelism according to the denoising discrepancy between these two paths. Our framework achieves $2.31\times$ and $2.07\times$ latency reductions on SDXL and SD3, respectively, using two NVIDIA RTX~3090 GPUs, while preserving image quality. This result confirms the generality of our approach across U-Net-based diffusion models and DiT-based flow-matching architectures. Our approach also outperforms existing methods in acceleration under high-resolution synthesis settings. Code is available at https://github.com/kaist-dmlab/Hybridiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。