揭示预测编码网络的稳定与高效机制，证明其优于传统反向传播。
Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networks
- 基于动力系统理论分析预测编码的稳定性与收敛性。
- 证明预测编码近似拟牛顿法，收敛更快且更稳定。
- 理论证明其比目标传播更接近拟牛顿更新，适合研究神经计算机制者。
基于能量的学习算法，如预测编码（PC），因其局部操作和生物可解释的误差修正机制而受到广泛关注。本文通过动力系统理论严格分析了预测编码的稳定性、鲁棒性和收敛性。首先，在损失函数和残差能量函数满足温和假设下，证明了预测编码具有李雅普诺夫稳定性，表明其在小随机扰动下具有内在鲁棒性，源于其明确的能量最小化动态。其次，正式建立预测编码更新近似于包含高阶曲率信息的拟牛顿方法，使其相比反向传播（BP）训练模型更稳定，收敛迭代次数更少。此外，基于该动力学框架，通过精确刻画高阶导数的作用，为预测编码与其它算法（如反向传播和目标传播）之间的相似性提供了新的理论界。这些界表明，预测编码在赫斯结构分析下显著更接近拟牛顿更新，深化了对预测编码相对于传统学习方法的稳定性与效率的理解。
原文摘要 · Abstract (English)
Energy-based learning algorithms, such as predictive coding (PC), have garnered significant attention in the machine learning community due to their theoretical properties, such as local operations and biologically plausible mechanisms for error correction. In this work, we rigorously analyze the stability, robustness, and convergence of PC through the lens of dynamical systems theory. We show that, first, PC is Lyapunov stable under mild assumptions on its loss and residual energy functions, which implies intrinsic robustness to small random perturbations due to its well-defined energy-minimizing dynamics. Second, we formally establish that the PC updates approximate quasi-Newton methods by incorporating higher-order curvature information, which makes them more stable and able to converge with fewer iterations compared to models trained via backpropagation (BP). Furthermore, using this dynamical framework, we provide new theoretical bounds on the similarity between PC and other algorithms, i.e., BP and target propagation (TP), by precisely characterizing the role of higher-order derivatives. These bounds, derived through detailed analysis of the Hessian structures, show that PC is significantly closer to quasi-Newton updates than TP, providing a deeper understanding of the stability and efficiency of PC compared to conventional learning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。