提出自适应误差阈值算法，显著提升混合精度深度学习的容错精度。
V-ABFT: Variance-Based Adaptive Threshold for Fault-Tolerant Matrix Multiplication in Mixed-Precision Deep Learning
- 基于方差建模验证差异，动态调整故障检测阈值。
- 相比现有方法误差阈值缩小6至48倍，误报率为零。
- 适用于多种精度，支持国产与通用硬件部署。
算法容错（ABFT）被广泛用于检测深度学习中矩阵乘法的隐性数据错误。然而，现有阈值确定方法存在严重问题：解析边界过于保守，而概率方法如A-ABFT的阈值比实际舍入误差大160–4200倍。本文提出V-ABFT，一种基于方差的自适应阈值算法，通过直接建模验证差异实现更紧的误差界。利用统计方差估计，V-ABFT将阈值与真实误差之比降至FP32/FP64下的7–20倍、BF16下的48–158倍，相较A-ABFT提升6–48倍，且在BF16、FP16、FP32、FP64下均保持零误报率。此外，我们证明，在融合核实现中于输出量化前验证时，低精度GEMM可采用FP32级阈值（$e_{ ext{max}} /approx 10^{-6}$），相比离线量化后验证（$e_{ ext{max}} /approx 10^{-3}$）实现约1000倍更细的检测粒度。实验复现了A-ABFT设置，并验证结果一致性。本方法仅需$O(n)$复杂度（使用最大/最小/均值统计），优于A-ABFT的$O(pn)$复杂度。在合成数据及真实模型权重（LLaMA-7B、GPT-2、ViT）上进行了广泛实验，证明V-ABFT对多种分布的有效性。该方法平台无关，已集成至NPU与GPU的容错GEMM实现中。
原文摘要 · Abstract (English)
Algorithm-Based Fault Tolerance (ABFT) is widely adopted to detect silent data corruptions (SDCs) in matrix multiplication, a cornerstone operation in deep learning systems. However, existing threshold determination methods face critical challenges: analytical bounds are overly conservative, while probabilistic approaches like A-ABFT yield thresholds $160$--$4200\times$ larger than actual rounding errors. We present V-ABFT, a variance-based adaptive threshold algorithm that achieves tighter error bounds by directly modeling the verification difference. By leveraging statistical variance estimation, V-ABFT reduces the threshold-to-actual-error ratio to approximately $7$--$20\times$ for FP32/FP64 and $48$--$158\times$ for BF16, representing a \textbf{6--48$\times$ improvement} over A-ABFT while maintaining zero false positive rate across BF16, FP16, FP32, and FP64 precisions. Furthermore, we demonstrate that for fused-kernel ABFT implementations that verify before output quantization, low-precision GEMM can use FP32-level thresholds ($e_{\max} \approx 10^{-6}$), enabling \textbf{$\sim$1000$\times$ finer detection granularity} compared to offline verification with low-precision output ($e_{\max} \approx 10^{-3}$). We reproduce A-ABFT's experimental setup and validate our implementation against the original paper's results. Our method requires only $O(n)$ complexity using max/min/mean statistics, compared to A-ABFT's $O(pn)$ for finding $p$ largest values. Extensive experiments on synthetic data and real model weights (LLaMA-7B, GPT-2, ViT) demonstrate V-ABFT's effectiveness across diverse distributions. V-ABFT is platform-agnostic and has been integrated into fault-tolerant GEMM implementations on both NPUs and GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。