让深度预测编码网络不再梯度消失，实现128层高效训练
Error Highways: Scaling Predictive Coding to Very Deep Networks
- 引入高速公路误差传播机制，直接连接输出误差与中间层
- 在MNIST和Fashion-MNIST上实现128层MLP的稳定训练，准确率不随深度下降
- 保持局部学习规则，适合生物可解释性神经网络研究者
预测编码网络（PCNs）提供了一种类脑、局部学习的替代反向传播方法。然而，其长期受限于浅层结构和简单基准测试。核心挑战在于学习信号从边界处传播时迅速衰减，导致内部层几乎不变。为此，我们提出高速公路误差传播（HEP），通过在预测编码的自由能函数中引入反馈矩阵 $V_{L o i}$，将选定隐藏状态直接耦合至被钳制的输出误差。该路径线性依赖隐藏状态，使每一步推理都获得与深度无关的校正量；而原生预测编码中输出误差到达第 $i$ 层时呈指数级衰减。此设计绕过雅可比链，同时保留局部更新规则。在MNIST与Fashion-MNIST上，HEP成功训练出高达128层的MLP，且准确率对深度具有鲁棒性。
原文摘要 · Abstract (English)
Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices $V_{L\to i}$ that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the $i$-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。