提出可扩展的硬件兼容梯度训练法,让大网络高效在硬件上训练。
Scaling of hardware-compatible perturbative training algorithms
- 用多重扰动法估算梯度,支持权重与节点双重扰动。
- 训练时间随网络增大反而下降,达目标精度更快。
- 兼容动量等优化器,适合未来类脑计算系统部署。
本文研究了多路梯度下降(MGD)这一可扩展且高效的零阶扰动训练方法,用于在硬件上估计损失函数梯度并进行随机梯度下降训练。该框架拓展至同时支持权重和节点扰动,分析其优劣。我们考察了使用MGD训练网络所需时间随网络规模和任务复杂度的变化。以往研究认为扰动训练难以扩展至大规模问题,因梯度估计时间与参数量线性相关。然而本工作表明,达到目标准确率所需时间并不呈线性增长,反而常随网络增大而减少。此外,我们证明MGD可作为梯度的即插即用替代方案,使动量等优化加速技术得以与之结合,确保与现有机器学习实践兼容。结果表明,MGD可在硬件上高效训练大型网络,性能接近反向传播,为未来类脑计算系统提供实用解决方案。
原文摘要 · Abstract (English)
In this work, we explore the capabilities of multiplexed gradient descent (MGD), a scalable and efficient perturbative zeroth-order training method for estimating the gradient of a loss function in hardware and training it via stochastic gradient descent. We extend the framework to include both weight and node perturbation, and discuss the advantages and disadvantages of each approach. We investigate the time to train networks using MGD as a function of network size and task complexity. Previous research has suggested that perturbative training methods do not scale well to large problems, since in these methods the time to estimate the gradient scales linearly with the number of network parameters. However, in this work we show that the time to reach a target accuracy--that is, actually solve the problem of interest--does not follow this undesirable linear scaling, and in fact often decreases with network size. Furthermore, we demonstrate that MGD can be used to calculate a drop-in replacement for the gradient in stochastic gradient descent, and therefore optimization accelerators such as momentum can be used alongside MGD, ensuring compatibility with existing machine learning practices. Our results indicate that MGD can efficiently train large networks on hardware, achieving accuracy comparable to backpropagation, thus presenting a practical solution for future neuromorphic computing systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。