arXiv:2506.23800cs.LG2025-06被引 6

解决深层预测编码网络训练失效问题,实现接近反向传播的性能。

Towards the Training of Deeper Predictive Coding Neural Networks

  • 设计精度加权优化,平衡各层误差分布
  • 新权重更新机制减少深层累积误差
  • 用辅助神经元控制残差路径能量传播速度

预测编码网络通过迭代能量最小化进行推理,具有时空局部性。然而在五到七层以上时性能显著下降。本文发现,这是由于权重更新中各层误差呈指数级失衡,且前层预测对深层更新无效所致。此外,使用跳连结构时,残差路径传递的能量比主路径更快,影响测试精度。为此,我们提出三种改进:一是精度加权的隐变量优化,平衡松弛阶段误差分布;二是新型权重更新机制,降低深层误差累积;三是引入辅助神经元,减缓残差连接中的能量传播速度。实验表明,这些方法使深层模型(如ResNet)性能接近反向传播,为复杂任务中的预测编码应用开辟新可能。

原文摘要 · Abstract (English)

Predictive coding networks are neural models that perform inference through an iterative energy minimization process, whose operations are local in space and time. While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers. In this work, we show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers. Furthermore, when training models with skip connections, the energy propagated by the residuals reaches higher layers faster than that propagated by the main pathway, affecting test accuracy. We address the first issue by introducing a novel precision-weighted optimization of latent variables that balances error distributions during the relaxation phase, the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers, and the third one by using auxiliary neurons that slow down the propagation of the energy in the residual connections. Empirically, our methods achieve performance comparable to backpropagation on deep models such as ResNets, opening new possibilities for predictive coding in complex tasks.

神经网络预测编码深度学习优化算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。