用随机无偏近似降低反向传播的计算与内存开销
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
- 用随机无偏近似替代精确的向量-雅可比乘积
- 在保持精度的前提下，显著减少每轮训练成本
- 适用于大规模神经网络训练，尤其适合资源受限场景
本文提出通过在反向传播中用随机无偏近似替代精确的向量-雅可比乘积，以降低深度神经网络训练的计算和内存开销。我们对达到目标精度所需训练轮数与每轮成本节省之间的权衡进行了理论分析，并识别出在稀疏性约束下具有最小方差最优性质的无偏估计方法。在多层感知机、BagNets 和视觉变换器等架构上的深入实验验证了理论结果，表明所提出的无偏随机反向传播方法在降低深度学习成本方面具有巨大潜力。
原文摘要 · Abstract (English)
In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations thereof during backpropagation. We provide a theoretical analysis of the trade-off between the number of epochs needed to achieve a target precision and the cost reduction for each epoch. We then identify specific unbiased estimates of vector-jacobian products for which we establish desirable optimality properties of minimal variance under sparsity constraints. Finally we provide in-depth experiments on multi-layer perceptrons, BagNets and Visual Transfomers architectures. These validate our theoretical results, and confirm the potential of our proposed unbiased randomized backpropagation approach for reducing the cost of deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。