用多方向切线近似梯度，提升训练效率与性能
Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
- 通过正交投影融合多个前向梯度，改进近似精度
- 增加切线数量显著提升优化效果，跨任务验证有效
- 适合追求高效、可并行训练的模型研究者
神经网络训练通常依赖反向传播计算梯度。尽管高效且精确，反向传播计算成本高、难以并行，且不符合生物合理性。前向梯度通过前向自动微分计算随机切线方向上的方向导数来近似梯度。以往研究多采用单个切线。本文深入分析多切线前向梯度，并提出基于正交投影的梯度融合方法。实验表明，增加切线数量可同时提升梯度近似质量与优化性能，在多种任务上均取得改善。
原文摘要 · Abstract (English)
The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。