系统分析大模型分布式并行策略，指导高效训练与推理设计。
Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
- 梳理集体通信与混合并行机制，建立数学理论框架。
- 提出计算与通信重叠的混合并行优化思路，提升部署效率。
- 提供主流架构案例参考，适合大模型系统开发者使用。
随着大语言模型（LLMs）的快速发展，大量方法被提出以在硬件设备间分布计算与内存，实现高效的训练与推理。尽管现有综述提供了技术概述，但对各类方法优劣的系统性分析，以及如何据此构建合理分布式系统的设计原则仍显不足。本文全面回顾了集体操作与分布式并行策略，辅以数学建模深化理论理解。进一步探讨混合并行设计，强调在训练与推理各阶段实现计算与通信的重叠。同时讨论利用成本模型自动搜索最优混合并行策略的最新进展。通过主流架构类别案例研究，揭示实证洞察，为研究人员与实践者选择并行策略提供指导。最后，指出当前大模型训练范式存在的开放挑战与局限，并展望下一代大规模模型发展的可能方向。
原文摘要 · Abstract (English)
With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。