不同大模型对系统设计需求差异大，3D/4D并行策略影响训练效率。
Comprehensive Performance Modeling and System Design Insights for Foundation Models
- 区分LLM与长序列科学模型，提出适配的3D/4D并行策略。
- 大语言模型在预训练阶段才显著依赖网络，对算力带宽要求较低。
- 揭示模型类型与系统特征的匹配关系，指导HPC系统设计。
生成式AI，特别是大型Transformer模型，正日益推动科学与工业领域的高性能计算（HPC）系统设计。本文分析了此类Transformer模型的性能特征，探讨其对模型类型、并行策略及HPC系统特性（加速器与互连网络）的敏感性。通过构建性能模型，我们探索了这一复杂设计空间并识别出关键组件。研究发现，不同Transformer类型在不同训练规模下需要不同的并行方式与系统配置。大语言模型在采用3D并行时表现优异，仅在预训练规模下显著增加网络需求，且对加速器容量和带宽依赖降低；而代表科学基础模型的长序列Transformer则对网络与存储容量有更均匀的依赖，需采用4D并行。分析强调了针对不同Transformer类型进行精细化性能建模的重要性，并指明了实现路径。代码已开源。
原文摘要 · Abstract (English)
Generative AI, in particular large transformer models, are increasingly driving HPC system design in science and industry. We analyze performance characteristics of such transformer models and discuss their sensitivity to the transformer type, parallelization strategy, and HPC system features (accelerators and interconnects). We utilize a performance model that allows us to explore this complex design space and highlight its key components. We find that different transformer types demand different parallelism and system characteristics at different training regimes. Large Language Models are performant with 3D parallelism and amplify network needs only at pre-training scales with reduced dependence on accelerator capacity and bandwidth. On the other hand, long-sequence transformers, representative of scientific foundation models, place a more uniform dependence on network and capacity with necessary 4D parallelism. Our analysis emphasizes the need for closer performance modeling of different transformer types keeping system features in mind and demonstrates a path towards this. Our code is available as open-source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。