提出无限宽网络下的局部学习参数化方法，揭示预测编码与目标传播的稳定机制。
Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
- 引入最大更新参数化（μP），在无限宽极限下分析局部学习算法。
- 发现预测编码在无限宽时趋近一阶梯度，目标传播偏好特征学习而非核方法。
- 该理论可跨模型宽度传递超参数，适合研究深度学习优化机制的研究者。
局部学习通过逐层局部目标和损失训练网络，被视为反向传播（BP）的替代方案。然而，由于局部性，其算法往往更复杂或需额外超参数，难以确定稳定运行的设置。为此，本文在无限宽极限下，针对两种代表性局部目标设计——预测编码（PC）与目标传播（TP）——引入最大更新参数化（μP），提供理论与量化洞察。验证了μP可在不同宽度模型间实现超参数迁移。进一步分析表明，μP具有传统BP所不具备的独特性质：在深层线性网络中，PC梯度介于一阶与高斯-牛顿型梯度之间，具体取决于参数化方式；在标准设置下，无限宽极限的PC行为更接近一阶梯度。对于TP，即使最后一层采用标准缩放（不同于经典μP），其局部损失优化仍倾向于特征学习而非核方法。
原文摘要 · Abstract (English)
Local learning, which trains a network through layer-wise local targets and losses, has been studied as an alternative to backpropagation (BP) in neural computation. However, its algorithms often become more complex or require additional hyperparameters because of the locality, making it challenging to identify desirable settings in which the algorithm progresses in a stable manner. To provide theoretical and quantitative insights, we introduce the maximal update parameterization ($μ$P) in the infinite-width limit for two representative designs of local targets: predictive coding (PC) and target propagation (TP). We verified that $μ$P enables hyperparameter transfer across models of different widths. Furthermore, our analysis revealed unique and intriguing properties of $μ$P that are not present in conventional BP. By analyzing deep linear networks, we found that PC's gradients interpolate between first-order and Gauss-Newton-like gradients, depending on the parameterization. We demonstrate that, in specific standard settings, PC in the infinite-width limit behaves more similarly to the first-order gradient. For TP, even with the standard scaling of the last layer, which differs from classical $μ$P, its local loss optimization favors the feature learning regime over the kernel regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。