TTrace通过对比张量差异，高效定位分布式训练中的隐性错误。
TTrace: Lightweight Error Checking and Diagnosis for Distributed Training
- 基于参考实现比对中间张量，系统化检测分布式训练错误。
- 在Megatron-LM中发现11个已知与3个新错误，仅需少于10行代码修改。
- 适用于低精度训练（BF16/FP8），已被主流开源框架采纳。
分布式训练是大规模神经网络模型（如大语言模型）跨数千个GPU扩展训练的关键技术。然而，分布式程序的复杂性使其极易出现无声错误——这类错误不产生明确信号，却导致训练结果错误。现有基于损失或梯度范数监控的调试方法间接、低效且无法定位问题。为此，我们设计并实现了TTrace，首个系统性差分测试框架，用于检测和定位分布式训练中的无声错误。TTrace将分布式训练的中间张量与可信参考实现的结果对齐。为准确比较浮点数值，我们提出一种新的数学分析方法，提供容差设置准则，可区分故障引起的误差与数值误差。实验表明，TTrace有效检测到Megatron-LM框架中的11个已有错误和3个新错误，仅需少于10行代码改动。该方法在多种训练方案中均有效，包括使用BF16和FP8的低精度训练。值得注意的是，一款流行的开源训练框架已将其方法集成至开发流程。
原文摘要 · Abstract (English)
Distributed training is essential for scaling the training of large neural network models, such as large language models (LLMs), across thousands of GPUs. However, the complexity of distributed training programs makes them particularly prone to silent bugs, which do not produce explicit error signals but lead to incorrect training outcomes. Effectively detecting and localizing such silent bugs in distributed training is challenging. Common debugging practices based on monitoring training loss or gradient norm curves are indirect, inefficient, and provide no way to localize bugs. To address those challenges, we design and implement TTrace, the first systematic differential testing system for detecting and localizing silent bugs in distributed training. TTrace aligns intermediate tensors from distributed training with those from a trusted reference implementation. To properly compare the floating-point values in the corresponding tensors, we propose a novel mathematical analysis that provides a guideline for setting tolerances, enabling TTrace to distinguish bug-induced errors from numerical errors. Experimental results demonstrate that TTrace effectively detects 11 existing bugs and 3 new bugs in the widely used Megatron-LM framework, while requiring fewer than 10 lines of code changes. TTrace is effective in various training recipes, including low-precision recipes involving BF16 and FP8. Notably, a popular open-source training framework has already adopted the method proposed by TTrace in its development workflow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。