用轻量级模型精准预测超大规模LLM训练时间
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
- 将LLM拆解为计算原语，逐层建模性能
- 128卡200亿参数模型预测误差仅4.98%~9.38%
- 纯CPU运行，适合快速调试硬件与训练策略
训练大语言模型是高性能计算中最具算力挑战的任务之一。由于Transformer组件、并行策略（数据、模型、流水线、张量）及多级通信间的复杂交互，预测跨数百块GPU分布的数十亿参数模型端到端训练时间仍具挑战。学习型模型需高成本采样，而解析型模型常难以应对真实网络与硬件复杂性。本文通过将LLM分解为核心计算原语，采用：(1) 操作级分解实现细粒度分析；(2) 基于轻量采样的硬件感知预测模型，针对关键操作建模；(3) 集成上述组件的端到端预测系统，覆盖复杂并行策略。关键在于，该方法已在两个超算系统上验证。框架对高达200亿参数模型在128块GPU上的预测平均误差为4.98%（Perlmutter，A100）和9.38%（Vista，GH200）。更重要的是，系统完全在CPU上运行，无需集群实验即可快速迭代硬件配置与训练策略。
原文摘要 · Abstract (English)
Training Large Language Models(LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end training time for multi-billion parameter models distributed across hundreds of GPUs remains challenging due to complex interactions between transformer components, parallelism strategies(data, model, pipeline, tensor), and multi-tier communication. Learned models require costly sampling, while analytical models often struggle with real-world network and hardware complexities. We address this by decomposing LLMs into core computational primitives and modeling them with: (1) operator-level decomposition for fine-grained analysis; (2) lightweight sampling based hardware-aware prediction models for key operations; (3) an end-to-end prediction system integrating these components across complex parallelization strategies. Crucially, our methodology has been validated on two large-scale HPC systems. Our framework achieves low average prediction errors-4.98\% on Perlmutter(A100) and 9.38\% on Vista(GH200)-for models up to 20B parameters across 128 GPUs. Importantly, it runs entirely on CPUs, enabling rapid iteration over hardware configurations and training strategies without costly on-cluster experimentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。