arXiv:2505.20892cs.LG2025-05被引 3

初始化时一次软对齐,让网络无需权重传输也能稳定训练

One-Time Soft Alignment Enables Resilient Learning without Weight Transport

  • 用一次初始软对齐替代反向传播的权重传输
  • 深度网络性能接近传统反向传播,且训练更稳定
  • 适合追求生物合理性与低资源消耗的深度学习研究

反向传播是深度学习的核心,但依赖对称权重传输和全局同步,计算开销大且不符合生物学规律。反馈对齐通过固定随机反馈近似误差梯度,避免了对称权重传输,但常面临学习性能差、不稳定的挑战,尤其在深层网络中。本文发现,仅在初始化阶段进行一次前向与反馈权重的软对齐,即可使深层网络达到与反向传播相当的性能,且学习过程中无需权重传输。这一简单初始化条件引导损失曲面中稳定误差最小化,提升网络可训练性。谱分析进一步显示,初始对齐促进更平滑的梯度流动,收敛至更平坦的极小值,从而提升泛化能力与鲁棒性。值得注意的是,允许适度偏离严格对称性反而比标准反向传播更具抗对抗攻击能力。这些发现表明,仅通过简单初始化策略,即可实现生物合理且资源高效的深度学习。

原文摘要 · Abstract (English)

Backpropagation is the cornerstone of deep learning, but its reliance on symmetric weight transport and global synchronization makes it computationally expensive and biologically implausible. Feedback alignment offers a promising alternative by approximating error gradients through fixed random feedback, thereby avoiding symmetric weight transport. However, this approach often struggles with poor learning performance and instability, especially in deep networks. Here, we show that a one-time soft alignment between forward and feedback weights at initialization enables deep networks to achieve performance comparable to backpropagation, without requiring weight transport during learning. This simple initialization condition guides stable error minimization in the loss landscape, improving network trainability. Spectral analyses further reveal that initial alignment promotes smoother gradient flow and convergence to flatter minima, resulting in better generalization and robustness. Notably, we also find that allowing moderate deviations from exact weight symmetry can improve adversarial robustness compared to standard backpropagation. These findings demonstrate that a simple initialization strategy can enable effective learning in deep networks in a biologically plausible and resource-efficient manner.

反向传播反馈对齐初始化生物合理性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。