arXiv:2607.16612stat.MLcs.LG2026-07

不靠反向传播训练模型主干,用分段前向梯度提升稳定性与效率。

Backpropagation-Free Trunk Training via the Split Forward Gradients

  • 将网络在中间表示处拆分,仅用雅可比-向量积估算主干梯度。
  • 1600万参数模型在WikiText-103上达困惑度387,优于冻结主干的668。
  • 无需反向传播,峰值内存降低35%,适合资源受限场景。

反向传播使深度网络训练内存开销大,因需存储中间激活值。前向模式方法虽避免此开销,但参数增多时梯度估计噪声加剧。本文提出分段前向梯度(Split-FG):在中间表示处拆分网络,精确计算输出头梯度,仅用雅可比-向量积估计主干梯度。该方法降低估计方差,无需通过主干的反向传播,且保持类似Adam的收敛性。实验发现,朴素前向梯度训练主干表现劣于随机初始化主干冻结的情况,可能因Adam对高噪声、欠定主干坐标更新过激。改用更小学习率后,1600万参数的GPT-2类模型在WikiText-103上验证困惑度降至387,优于冻结主干的668和纯前向梯度基线的2885(反向传播为150)。Split-FG在表格数据集上表现最优,在CIFAR-10上达60.5%准确率,CIFAR-100上达35.2%,使用重头设计。相比匹配的反向传播,峰值内存最多减少35%,但随前向模式主干增大,性能差距扩大。

原文摘要 · Abstract (English)

Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a $16$M-parameter GPT-2-style model reaches validation perplexity $387$, compared with $668$ for the frozen-trunk control and $2{,}885$ for a matched pure forward-gradient baseline (backpropagation reaches $150$). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches $60.5\%$ on CIFAR-10 and $35.2\%$ on CIFAR-100 with a heavy-head design. It reduces peak memory by up to $35\%$ relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.

前向梯度内存优化无反向传播高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。