用局部更新逼近反向传播，让神经网络训练更快更准
Augmented Lagrangian Predictive Coding
- 通过局部拉格朗日乘子累积误差，实现每层独立更新却逼近反向传播
- 在深度128的非线性网络中性能媲美反向传播，尤其改善了窄深网络表现
- 揭示了分布式系统如何通过局部动态传递精准梯度信号
预测编码（PC）是一种替代反向传播（BP）的局部学习方法，通过局部能量最小化动力学训练深层网络，而非全局反向传播。本文提出增强拉格朗日预测编码（PC-ALM），在保持PC推理预算的同时，通过将每层约束误差累积为层内拉格朗日乘子，使权重更新趋近于BP。在线性PC网络中，PC-ALM可收敛至与BP梯度精确一致的平衡点，且仅通过层内更新实现。在深度达128的非线性网络中，其性能全面匹配BP，尤其在深度窄网络中显著优于传统PC。PC-ALM引入层内递归动态，其演化由增广拉格朗日的对偶上升驱动，而非PC的标量能量热流。我们观察到‘弹道式’信用传播——信用信号在极深网络中均匀分布，远快于PC的缓慢扩散传播。该框架不仅改进了算法，也为分布式系统如何通过纯局部动力学计算和传播类反向传播信号提供了新视角。
原文摘要 · Abstract (English)
Predictive coding (PC) is a local-learning alternative to backpropagation (BP), training deep networks via local energy-minimization dynamics rather than a global backward pass. We introduce Augmented Lagrangian Predictive Coding (PC-ALM), which maintains PC's inference budget but aligns each weight update toward BP by accumulating per-layer constraint errors into a layer-local Lagrange multiplier. In linear PC networks, PC-ALM converges to an equilibrium with exact BP gradients distributed across the network via only layer-local updates. We analyze PC-ALM in nonlinear PC networks up to depth 128 and show that it matches BP performance across all width-depth regimes, notably in deep narrow networks where PC underperforms. PC-ALM introduces recurrent dynamics in each layer's activations. Compared to PC's heat flow on a scalar energy, PC-ALM dynamics are driven by dual ascent on the augmented Lagrangian. We observe "ballistic" credit propagation across very deep networks, with credit signals evenly distributed across layers, compared to PC's slow, diffusive credit propagation. Beyond the algorithm itself, the augmented Lagrangian framework offers a generalization of PC, and may yield insights into how distributed systems could compute and propagate BP-like credit signals through purely local dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。