用近似梯度替代反向传播，降低训练成本且无偏差。
Linear Gradient Prediction with Control Variates
- 用控制变量法构造无偏梯度估计，避免完整反向传播。
- 基于神经正切核理论设计梯度预测器，提升更新效率。
- 在视觉Transformer上验证有效，显著降低训练开销。
我们提出一种新的神经网络训练方法，旨在降低训练成本。该方法使用近似预测梯度代替需要昂贵反向传播的完整梯度。我们推导出一种基于控制变量的技术，确保更新为真实梯度的无偏估计。此外，我们提出一种受神经正切核理论启发的新梯度预测器构造方法。我们在视觉Transformer分类任务上实证验证了该技术的有效性。
原文摘要 · Abstract (English)
We propose a new way of training neural networks, with the goal of reducing training cost. Our method uses approximate predicted gradients instead of the full gradients that require an expensive backward pass. We derive a control-variate-based technique that ensures our updates are unbiased estimates of the true gradient. Moreover, we propose a novel way to derive a predictor for the gradient inspired by the theory of the Neural Tangent Kernel. We empirically show the efficacy of the technique on a vision transformer classification task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。