通过迭代推理验证分布式模型输出是否可还原，定位训练错误。
Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference
- 用迭代重写技术静态验证分布式模型能否还原原始顺序输出
- 在GPT和Llama-3上验证，可处理大规模模型且支持实际部署
- 输出可定位具体出错位置，适合调试分布式训练代码的开发者
当前大型模型因计算与内存需求超出单个GPU能力，普遍采用分布式训练与推理。程序员将顺序模型通过分片策略分布到多个GPU上执行，但此过程易引入错误，导致分布式模型输出与原始顺序模型不一致。本文提出一种静态检测方法，通过检查模型精炼性——即能否从分布式模型输出重构出顺序模型输出——来识别此类错误。该方法在GraphGuard中实现，利用迭代重写证明模型精炼性。实验在GPT和Llama-3上进行，验证了其对现代大模型及真实部署场景的可扩展性，并提供可操作的定位信息，帮助快速修复问题。
原文摘要 · Abstract (English)
Distributed machine learning training and inference is common today because today's large models require more memory and compute than can be provided by a single GPU. Distributed models are generally produced by programmers who take a sequential model specification and apply several distribution strategies to distribute state and computation across GPUs. Unfortunately, bugs can be introduced in the process, and a distributed model implementation's outputs might differ from the sequential model's outputs. In this paper, we describe an approach to statically identify such bugs by checking model refinement, that is, can the sequential model's outputs be reconstructed from the distributed model's outputs? Our approach, implemented in GraphGuard, uses iterative rewriting to prove model refinement. Our approach can scale to today's large models and deployments: we evaluate it using GPT and Llama-3. Further, it provides actionable output that aids in bug localization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。