用少量显卡模拟数千张卡的大模型训练,精准还原性能与内存行为。
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM

- 通过切片构建高保真执行图,捕捉大规模计算通信依赖。
- 迭代时间误差仅0.58%,峰值显存误差低于0.01%。
- 适合调试、优化训练框架的工程师,无需占用真实大集群。
当前大语言模型训练依赖上千张GPU组成的集群,导致开发、调试和性能调优成本高昂。工程师常需复现生产环境行为,但大规模集群资源紧张且难以独占。现有仿真依赖复杂性能模型,难以维护;降规模实验又无法反映尺度相关行为。本文提出PrismLLM,通过基于切片的高保真执行图构建,实现计算、通信与依赖关系的精确捕获。采用混合仿真策略:部分真实运行,其余以虚拟角色重放。在大规模LLM训练任务上验证,PrismLLM平均迭代时间误差仅为0.58%,峰值显存误差小于0.01%。可使用少于1%的物理GPU,模拟高达8192张卡的集群行为。
原文摘要 · Abstract (English)
Large language model (LLM) training today runs on clusters spanning thousands of GPUs. While this scale enables rapid model advances, developing, debugging, and performance-tuning the training framework inevitably becomes complex and costly. This is because engineers often need to reproduce production behaviors to diagnose failures or evaluate optimizations, thereby demanding frequent and even exclusive access to production-scale clusters -- which becomes increasingly hard given that the majority of GPUs are already committed to production workloads. Simulation relies on complex performance models that are difficult to maintain, and downscaled experiments often fail to capture scale-dependent behaviors. We present PrismLLM to decouple large-scale execution from the need to access large clusters, enabling engineers to run and observe ranks of interest under faithful large-scale behavior using only a few GPUs. PrismLLM constructs a high-fidelity execution graph via a slicing-based approach that captures computation, communication, and dependencies of the target scale. Then, PrismLLM performs hybrid emulation where selected ranks execute the original program while the remaining ranks are replayed as virtual participants. Experiments on large-scale LLM training workloads show that PrismLLM accurately reproduces performance and memory behavior, achieving only 0.58\% average error in iteration time and less than 0.01\% error in peak GPU memory usage. PrismLLM can emulate clusters of up to 8192 GPUs using fewer than 1\% of the physical GPUs required by the original deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。