发现GPU矩阵乘法的浮点误差并非随机噪声，而是有结构的协同扰动。
On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication
- 通过对比单输入与批处理结果，发现误差高度相关而非独立
- float16下近50%误差方差来自非对角项，表明噪声具方向性
- 挑战了传统随机噪声假设，适合关注模型可靠性研究者
浮点数不满足结合律，导致GPU上的深度学习基础运算（如矩阵乘法）本质上具有非确定性。尽管如此，由此产生的数值误差的统计特性仍不清楚。普遍假设认为这些误差服从独立同分布（i.i.d.）高斯噪声。本文通过实证检验该假设，发现其无法描述真实GPU行为。通过比较单输入与批处理矩阵乘法输出，发现虽i.i.d.模型预测存在非零输出不稳定，但实际预测翻转率为0.00%。通过协方差分析，揭示原因在于浮点误差具有结构性且高度相关：对于float16，近50%的总误差方差存在于非对角项中，表明噪声表现为协调的、有方向的扰动，而非随机静态。这一发现挑战了现有对数值噪声的随机性认知，并为在硬件非确定性下分析深度学习可靠性提供了理论基础。
原文摘要 · Abstract (English)
Floating-point non-associativity makes fundamental deep learning operations, such as matrix multiplication (matmul) on GPUs, inherently non-deterministic. Despite this, the statistical structure of the resulting numerical error remains poorly understood. A common working assumption is that these errors behave as independent and identically distributed (i.i.d.) Gaussian noise. In this paper, we empirically test this assumption and show that it fails to describe real GPU behavior. By comparing outputs of single-input and batched matmuls, we find that while the i.i.d. model predicts non-zero output instability, empirical results show a 0.00% prediction flip rate. Through covariance analysis, we uncover the cause: the floating-point error is structured and highly correlated. For float16, nearly 50% of the total error variance lies in off-diagonal terms, revealing that the noise behaves as a coordinated, directional perturbation rather than random static. This result challenges the prevailing stochastic view of numerical noise and provides a principled foundation for analyzing deep learning reliability under hardware non-determinism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。