提出无需反向传播的梯度估计方法，降低偏差与方差，可扩展至更大网络。
Towards Scalable Backpropagation-Free Gradient Estimation
- 通过操纵上游雅可比矩阵设计梯度猜测方向，减少估计偏差与方差。
- 在宽网络下表现更好，随着网络宽度增加性能提升。
- 揭示了梯度低维结构与误差关系，适合研究高效训练机制者参考。
尽管反向传播（backpropagation）在深度学习中极为成功，但其需要网络的前向和反向两次遍历，并存储中间激活值。现有基于前向模式自动微分的梯度估计方法因估计方差过高，难以扩展至大型网络。此前缓解方差的努力引入了显著偏差，削弱了估计有效性。本文提出一种新梯度估计方法，通过在计算猜测方向时操作上游雅可比矩阵，同时降低偏差与方差。该方法展现出良好前景，且具备向更大网络扩展的潜力，甚至在网络宽度增加时表现更优。我们通过偏差与方差的分析，揭示了神经网络梯度的低维结构特征，深化了对该方法的理解。
原文摘要 · Abstract (English)
While backpropagation--reverse-mode automatic differentiation--has been extraordinarily successful in deep learning, it requires two passes (forward and backward) through the neural network and the storage of intermediate activations. Existing gradient estimation methods that instead use forward-mode automatic differentiation struggle to scale beyond small networks due to the high variance of the estimates. Efforts to mitigate this have so far introduced significant bias to the estimates, reducing their utility. We introduce a gradient estimation approach that reduces both bias and variance by manipulating upstream Jacobian matrices when computing guess directions. It shows promising results and has the potential to scale to larger networks, indeed performing better as the network width is increased. Our understanding of this method is facilitated by analyses of bias and variance, and their connection to the low-dimensional structure of neural network gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。