xdiT通过混合并行提升DiT推理速度，支持大规模部署
xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
- 采用序列并行与新型分块流水线并行结合
- 在以太网连接的8卡节点上实现DiT模型可扩展推理
- 适合需要实时生成高分辨率图像的AI系统研发者
扩散模型在生成高质量图像和视频中起关键作用。受OpenAI Sora成功启发，扩散模型的核心结构正从U-Net转向Transformer，即扩散Transformer（DiTs）。然而，生成高质量内容需更长序列长度，导致注意力机制计算量呈指数增长，显著增加DiTs推理延迟。实时部署要求并行推理，但单一并行方法在大规模下扩展性差。本文提出xdiT，一种全面的DiTs并行推理引擎。经深入分析现有并行方法，xdiT选用序列并行（SP）和一种新型分块级流水线并行（PipeFusion）作为图像内并行策略，辅以条件引导（CFG）并行实现图像间并行。xdiT支持灵活混合多种并行方式，提供强健且可扩展的解决方案。在两台8xL40 GPU（PCIe）节点（以太网互联）及一台8xA100（NVLink）节点上的实验表明，xdiT在五种前沿DiTs模型上均展现卓越扩展性。首次在以太网连接的GPU集群上验证了DiTs的可扩展性。代码已开源：https://github.com/xdit-project/xDiT。
原文摘要 · Abstract (English)
Diffusion models are pivotal for generating high-quality images and videos. Inspired by the success of OpenAI's Sora, the backbone of diffusion models is evolving from U-Net to Transformer, known as Diffusion Transformers (DiTs). However, generating high-quality content necessitates longer sequence lengths, exponentially increasing the computation required for the attention mechanism, and escalating DiTs inference latency. Parallel inference is essential for real-time DiTs deployments, but relying on a single parallel method is impractical due to poor scalability at large scales. This paper introduces xDiT, a comprehensive parallel inference engine for DiTs. After thoroughly investigating existing DiTs parallel approaches, xDiT chooses Sequence Parallel (SP) and PipeFusion, a novel Patch-level Pipeline Parallel method, as intra-image parallel strategies, alongside CFG parallel for inter-image parallelism. xDiT can flexibly combine these parallel approaches in a hybrid manner, offering a robust and scalable solution. Experimental results on two 8xL40 GPUs (PCIe) nodes interconnected by Ethernet and an 8xA100 (NVLink) node showcase xDiT's exceptional scalability across five state-of-the-art DiTs. Notably, we are the first to demonstrate DiTs scalability on Ethernet-connected GPU clusters. xDiT is available at https://github.com/xdit-project/xDiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。