用极少随机位实现随机舍入，避免计算偏差。
On Stochastic Rounding with Few Random Bits
- 仅用少量随机位实现随机舍入，降低计算开销。
- 发现部分简化实现会引入显著偏差，影响结果准确性。
- 适用于低精度计算中的机器学习场景，需注意配置参数。
大规模数值计算越来越多地采用低精度（LP）浮点格式和混合精度算术，可通过随机舍入（SR）技术进一步提升性能，即根据中间高精度值与两个舍入候选值的距离，随机向上或向下舍入。然而，随机舍入需要额外的随机比特源，而高质量随机比特的生成会带来额外计算成本。因此，如何在保持随机舍入理想性质的前提下，尽可能减少所需随机比特数量，成为关键问题。本文研究了多种少比特随机舍入（FBSR）的实现方式，发现若干常见简化方法可能引入显著偏差，这些偏差在无限比特、无限精度分析中并不存在。论文通过机器学习实例验证了这些偏差的影响，提醒从业者在设计或采用低精度浮点计算时，需关注此类配置参数。代码已开源：http://github.com/graphcore-research/arith25-stochastic-rounding。
原文摘要 · Abstract (English)
Large-scale numerical computations make increasing use of low-precision (LP) floating point formats and mixed precision arithmetic, which can be enhanced by the technique of stochastic rounding (SR), that is, rounding an intermediate high-precision value up or down randomly as a function of the value's distance to the two rounding candidates. Stochastic rounding requires, in addition to the high-precision input value, a source of random bits. As the provision of high-quality random bits is an additional computational cost, it is of interest to require as few bits as possible while maintaining the desirable properties of SR in a given computation, or computational domain. This paper examines a number of possible implementations of few-bit stochastic rounding (FBSR), and shows how several natural implementations can introduce sometimes significant bias into the rounding process, which are not present in the case of infinite-bit, infinite-precision examinations of these implementations. The paper explores the impact of these biases in machine learning examples, and hence opens another class of configuration parameters of which practitioners should be aware when developing or adopting low-precision floating point. Code is available at http://github.com/graphcore-research/arith25-stochastic-rounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。