探究前向-前向算法训练中各层精度变化规律
The Forward-Forward Algorithm: Characterizing Training Behavior
- 用两次前向传播替代反向传播，每层独立优化局部损失
- 深层网络层精度提升滞后于浅层，且浅层精度主导整体表现
- 适合关注神经网络内部训练机制的科研人员
前向-前向算法是一种替代反向传播的学习方法，通过两次前向传播而非前向加反向传播实现。该算法使用层内局部损失函数，根据每次前向传播的层激活值进行优化，而非依赖单一全局目标函数。本文通过分析训练过程中模型与各层准确率的变化动态，深入理解其内部行为机制。研究考察了不同系统特性对层与整体模型准确率的影响，揭示了层深度对准确率提升速度的影响，以及各层准确率与整体模型准确率之间的相关性。实证结果表明，前向-前向网络中深层层的准确率提升存在延迟，而浅层准确率与整体模型准确率强相关。
原文摘要 · Abstract (English)
The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。