提升扩散模型分布式推理效率,突破多卡协同瓶颈。
SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
- 根据GPU网络拓扑优化序列并行通信模式。
- 实现跨节点全归约操作与计算重叠,提速1.35倍。
- 采用单边通信减少同步开销,适合大规模生成任务。
扩散Transformer(DiTs)在高质量图像与视频生成中应用日益广泛。随着对高分辨率图像和长视频需求增长,单卡推理因延迟升高和激活值过大而效率低下。现有框架采用序列并行(SP)技术如Ulysses Attention和Ring Attention进行扩展,但存在三方面局限:(1) 未适配现代GPU机群的网络拓扑,通信效率低;(2) 跨节点通信中所有到所有(all-to-all)操作造成延迟瓶颈;(3) 使用双端通信库导致GPU发送-接收同步与计算开销大。为此,我们提出StreamFusion,一种面向拓扑感知的高效DiT服务引擎。其包含三项创新:(1) 拓扑感知序列并行技术,兼顾机间与机内带宽差异;(2) Torus Attention,一种新式SP方法,支持跨节点all-to-all操作与计算重叠;(3) 单边通信实现,显著降低发送-接收同步与计算开销。实验表明,StreamFusion相比当前最优方案平均提速1.35倍(最高达1.77倍)。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have gained increasing adoption in high-quality image and video generation. As demand for higher-resolution images and longer videos increases, single-GPU inference becomes inefficient due to increased latency and large activation sizes. Current frameworks employ sequence parallelism (SP) techniques such as Ulysses Attention and Ring Attention to scale inference. However, these implementations have three primary limitations: (1) suboptimal communication patterns for network topologies on modern GPU machines, (2) latency bottlenecks from all-to-all operations in inter-machine communication, and (3) GPU sender-receiver synchronization and computation overheads from using two-sided communication libraries. To address these issues, we present StreamFusion, a topology-aware efficient DiT serving engine. StreamFusion incorporates three key innovations: (1) a topology-aware sequence parallelism technique that accounts for inter- and intra-machine bandwidth differences, (2) Torus Attention, a novel SP technique enabling overlapping of inter-machine all-to-all operations with computation, and (3) a one-sided communication implementation that minimizes GPU sender-receiver synchronization and computation overheads. Our experiments demonstrate that StreamFusion outperforms the state-of-the-art approach by an average of $1.35\times$ (up to $1.77\times$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。