通过重复采样提升前向梯度下降收敛速度，消除其与随机梯度下降的性能差距。
Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling
- 对每个样本执行多次前向梯度步，缓解传统前向梯度下降的慢收敛问题。
- 当重复次数ℓ≥d时，误差率逼近标准随机梯度下降，理论优势显著。
- 适用于有低维结构输入数据的场景，尤其适合生物可解释性模型设计。
前向梯度下降（FGD）因其无需反向传播而被认为比标准梯度下降更具生物学合理性。然而，针对具有d个参数的线性模型，已有研究发现其预测误差比随机梯度下降（SGD）慢一个因子d。本文证明：若对每个训练样本计算ℓ次FGD步骤，则该子优性因子可降至d/(ℓ ∧ d)；当ℓ ≳ d时，该差距消失。此外，我们还发现重复采样的FGD能自适应输入分布中的低维结构。核心数学挑战在于控制重复采样带来的依赖关系。
原文摘要 · Abstract (English)
Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with $d$ parameters, previous work has found that the prediction error of FGD is, however, by a factor $d$ slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing $\ell$ FGD steps based on each training sample, this suboptimality factor becomes $d/(\ell \wedge d)$ and thus the suboptimality of the rate disappears if $\ell \gtrsim d.$ We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。