提出新型数值计算框架,解决超大动态范围计算中的溢出问题。
Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation
- 引入广义数量级(GOOMs),扩展传统浮点数表示能力。
- 在GPU上实现并行前缀扫描,支持超长序列复杂数值运算。
- 可稳定计算矩阵乘积、李雅普诺夫指数谱及深层循环网络依赖关系。
从深度学习到金融建模等多个领域,长期序列的实数复合计算常导致灾难性数值下溢或上溢。本文提出广义数量级(GOOMs),作为传统数量级的合理延伸,将浮点数视为特例,在实践中实现了比以往更大幅度的数值动态范围稳定计算。我们实现了基于高效自定义并行前缀扫描的GOOMs,支持在如GPU等并行硬件上的原生执行。通过三项代表性实验验证:(1) 在远超标准浮点数极限的情况下进行实数矩阵乘积复合;(2) 并行估计李雅普诺夫指数谱,速度较旧方法提升多个数量级,并采用新颖的选择性重置方法防止状态共线;(3) 以非对角递归状态并行计算深层循环神经网络中的长程依赖,无需任何稳定性机制。结果表明,结合高效并行扫描的GOOMs为高动态范围应用提供了可扩展且数值鲁棒的替代方案。
原文摘要 · Abstract (English)
Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that incorporates floating-point numbers as a special case, and which in practice enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments, all of which were previously considered impractical or impossible, and now become possible and practical: (1) compounding real matrix products far beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, orders of magnitude faster than with previous methods, applying a novel selective-resetting method to prevent state colinearity; and (3) capturing long-range dependencies in deep recurrent neural networks with non-diagonal recurrent states, computed in parallel via a prefix scan, without requiring any form of stabilization. Our results show that our implementation of GOOMs, combined with efficient parallel scanning, offers a scalable and numerically robust alternative to conventional floating-point numbers for high-dynamic-range applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。