受生物突触启发，提出乘法更新算法实现低精度稳定训练
Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks
- 基于对数正态分布设计乘法更新机制，结合噪声与正则化
- 在视觉变换器和GPT-2上实现低精度前向计算的稳定训练
- 适合追求能效比的边缘设备部署，尤其关注低精度推理
神经科学研究表明，生物突触遵循对数正态分布，其变化可由噪声乘法动力学解释。生物网络即使在突触传输不可靠导致动态波动的条件下也能稳定运行。本文提出：能否在人工神经网络中设计类似的乘法训练机制？为此，我们推导出一种贝叶斯学习规则，假设权重后验服从对数正态分布，从而提出新的对数正态乘法动力学（LMD）算法。该算法采用乘法更新，并同时施加噪声与正则化。方法实现简单，仅需额外存储一个向量，与Adam相当易用。实验表明，LMD可在低精度前向操作下实现视觉变换器和GPT-2的从头稳定且高精度训练。结果提示，这种生物特性驱动的乘法动力学或可支持未来节能硬件上的稳定低精度推理与学习。
原文摘要 · Abstract (English)
Studies in neuroscience have shown that biological synapses follow a log-normal distribution whose transitioning can be explained by noisy multiplicative dynamics. Biological networks can function stably even under dynamically fluctuating conditions arising due to unreliable synaptic transmissions. Here we ask: Is it possible to design similar multiplicative training in artificial neural networks? To answer this question, we derive a Bayesian learning rule that assumes log-normal posterior distributions over weights which gives rise to a new Log-Normal Multiplicative Dynamics (LMD) algorithm. The algorithm uses multiplicative updates with both noise and regularization applied multiplicatively. The method is as easy to implement as Adam and only requires one additional vector to store. Our results show that LMD achieves stable and accurate training-from-scratch under low-precision forward operations for Vision Transformer and GPT-2. These results suggest that multiplicative dynamics, a biological feature, may enable stable low-precision inference and learning on future energy-efficient hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。