提出工具自动检测深度学习算子的数值不稳定性。
Automated Numerical Stability Analysis of Deep Learning Operators
- 基于随机验证与数据扰动，单次计算即可完成数值稳定性检测。
- 能定位不稳定的算子并支持训练推理全程监控。
- 适合关注数值稳定性的模型开发者与系统优化研究者。
有限精度算术不可避免地引入数值近似误差。数值计算可能因精度不足或公式不当导致数值不稳定。本文提出一种统一的软件工具，用于深度学习算子的随机数值验证。该工具在支持的操作上遵循CESTAC方法，并对GEMM类核函数采用算子级数据扰动近似。所开发的软件不仅能通过单次计算实现数值验证，还可识别数值不稳定的根源，并在深度学习训练与推理过程中提供数值稳定性监控。我们在多种任务中验证了其对注入不稳定性算子的检测有效性。我们认为该方法与工具为构建数值稳定的计算核提供了重要洞察，尤其对高效且稳定的深度学习训练与推理至关重要。
原文摘要 · Abstract (English)
Finite-precision arithmetic unavoidably introduces numerical approximation errors. Numerical computations may use insufficient precision or an improper formulation, which leads to numerical instability. In this paper, we introduce a unified software tool for stochastic numerical validation of deep-learning operators. The tool follows CESTAC on supported exposed operations and uses an operator-level data-perturbation approximation for GEMM-like kernels. Our developed software not only enables numerical validation with a single computation pass but also detects the sources of numerical instability and provides numerical stability monitoring during deep learning training and inference. We verified its effectiveness on the detection of polluted operators with injected numerical instabilities across various tasks. We believe that our developed method and tools provide valuable insights into developing numerically stable computing kernels, which are particularly critical for numerically stable and efficient deep learning training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。