随机舍入能提升矩阵奇异值，增强模型稳定性。
Stochastic Rounding Increases Small Singular Values
- 提出随机舍入可作为谱正则化器，作用于奇异值分布尾部。
- 发现其不仅提升最小奇异值，还整体抬升尾部奇异值簇。
- 适用于低精度计算和机器学习中的数值稳定场景。
过去六年间，随机舍入（SR）作为低精度浮点计算的量化方案重新受到关注，广泛应用于数值分析与现代机器学习系统。近期研究发现，SR通过增加极细长（或对称地极短宽）矩阵的最小奇异值，起到隐式正则化作用。本文在两个方向上显著深化并拓展了这一理解：首先，证明了SR的正则化效应不局限于极端长宽比情形，即使在常数长宽比下依然有效；其次，揭示了SR并非仅调节最小奇异值，而是整体提升奇异值谱尾部的多个奇异值簇。这些结果为随机舍入提供了更普遍的谱正则化刻画，表明其影响范围超越极端长宽比，并作用于更广泛的奇异值谱区域。
原文摘要 · Abstract (English)
Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications spanning numerical analysis and modern machine learning systems. Recent work has shown that SR acts as an implicit regularizer by increasing the smallest singular value of extremely tall-and-thin (or, symmetrically, short-and-fat) matrices. In this work, we substantially sharpen and extend this understanding in two directions. First, we show that the regularization effect of SR is not restricted to extreme aspect ratio regimes: it persists for matrices with constant aspect ratio. Second, we demonstrate that SR does not merely regularize the smallest singular value, but instead lifts entire clusters of singular values at the tail of the spectrum. Together, these results provide a more general characterization of stochastic rounding as a spectral regularizer, revealing that its effects extend beyond extremal aspect ratios and act on a broader portion of the singular value spectrum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。