改进初始化方法，让预测编码网络训练更快更准。
Faster Predictive Coding Networks via Better Initialization
- 通过优化神经元初始化，减少迭代计算次数。
- 在监督与无监督任务中均显著加快收敛速度。
- 适合研究高效神经网络训练机制的学者参考。
旨在扩展受神经科学启发的学习算法以适应大规模神经网络的研究正加速推进。近期，能量基学习算法（如预测编码）因其通用性和数学基础成为关键研究方向。然而，这类方法因迭代特性导致计算开销大，限制了其应用。本文表明，预测编码网络中神经元的初始化选择至关重要，可显著缩短训练时间。为此，我们提出一种新初始化技术，旨在保留先前训练样本的迭代进展。实验结果表明，在监督与无监督设置下，该方法均大幅提升了收敛速度并降低了最终测试损失，为弥合预测编码与反向传播在计算效率与性能间的差距提供了可行路径。
原文摘要 · Abstract (English)
Research aimed at scaling up neuroscience inspired learning algorithms for neural networks is accelerating. Recently, a key research area has been the study of energy-based learning algorithms such as predictive coding, due to their versatility and mathematical grounding. However, the applicability of such methods is held back by the large computational requirements caused by their iterative nature. In this work, we address this problem by showing that the choice of initialization of the neurons in a predictive coding network matters significantly and can notably reduce the required training times. Consequently, we propose a new initialization technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples. Our approach suggests a promising path toward reconciling the disparities between predictive coding and backpropagation in terms of computational efficiency and final performance. In fact, our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。