提出μPC方法,让预测编码可训练128层深度网络。
$μ$PC: Scaling Predictive Coding to 100+ Layer Networks
- 用深度μP参数化改进预测编码,解决深层网络训练难题。
- 128层残差网络在分类任务上表现媲美基准,调参极少。
- 支持权重与激活学习率零样本跨宽度深度迁移,适合脑启发算法研究者。
反向传播(BP)的生物学不合理性促使人们探索基于局部信息的类脑算法,如预测编码(PC)和平衡传播。然而,这些算法长期难以训练极深网络,无法在大规模场景中与BP竞争。近期,有研究将扩展PC网络(PCNs)列为社区挑战(Pinchetti等,2024)。本文通过采用深度μP参数化(Yang等,2023;Bordelon等,2023),提出名为μPC的方法,成功实现100+层PCN的稳定训练。分析表明,标准PCN在深层时存在多种训练病态。尽管μPC仅缓解部分不稳定性,仍能以极少调参实现128层残差网络在简单分类任务上的稳定训练,并达到与当前基准相当的性能。此外,μPC支持权重与活动学习率在不同宽度和深度间的零样本迁移。结果为扩展预测编码至复杂架构迈出第一步,对其他局部算法也有启示。μPC代码已作为JAX库开源。
原文摘要 · Abstract (English)
The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community (Pinchetti et al., 2024). Here, we show that 100+ layer PCNs can be trained reliably using a Depth-$μ$P parameterisation (Yang et al., 2023; Bordelon et al., 2023) which we call "$μ$PC". By analysing the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, $μ$PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, $μ$PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC to more complex architectures and have implications for other local algorithms. Code for $μ$PC is made available as part of a JAX library for PCNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。