arXiv:2605.06240cs.LGcs.AI2026-05

FF网络中深层特征分离受早期层影响,修复后性能提升但不影响最终准确率。

Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

  • 提出累积优劣导致的层间依赖现象,即越深层梯度衰减越严重
  • 三种局部修复方法使深层分离度提升4至45倍,准确率变化小于1%
  • 适用于优化诊断与模型调试,尤其适合研究前向传播训练机制

前向-前向(FF)训练使每层基于局部优劣标准学习。然而在累积优劣变体中,后续层会继承前期层已部分完成的任务。我们将其形式化为层自由骑乘:在软正弦准则下,到达第d层的类别判别梯度随前序层积累的正边际呈指数衰减。研究了三种局部修复方法——逐块、硬度门控和深度缩放——无需依赖反向传播梯度即可恢复当前层分离度。在CIFAR-10和CIFAR-100上,这些方法使深层分离统计显著改善,深层分离度提升4×至45×,而准确率变化低于1个百分点。在Tiny ImageNet上的跨数据集验证也揭示了层健康诊断与最终准确率间的定性差距。校准实验进一步表明,架构与增强策略对最终准确率的影响大于本文所研究的训练规则修改。因此,累积自由骑乘是真实存在且可修复的优化病态,但在所研究的训练规则、架构与数据集下,并非限制准确率的主导因素。

原文摘要 · Abstract (English)

Forward-Forward (FF) training allows each layer to learn from a local goodness criterion. In cumulative-goodness variants, however, later layers can inherit a task that earlier layers have already partially separated. We formalize this phenomenon as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block $d$ decays exponentially with the positive margin accumulated by preceding blocks. We then study three local remedies -- per-block, hardness-gated, and depth-scaled -- that recover current-layer separation measures without relying on backpropagated gradients. On CIFAR-10 and CIFAR-100, these remedies dramatically improve layer-separation statistics, with $4\times$--$45\times$ gains in deeper layers, while changing accuracy by less than one percentage point for non-degenerate training procedures. Tiny ImageNet provides a tougher cross-dataset check for our selected block-wise configuration and reveals the same qualitative gap between layer-health diagnostics and final accuracy. Calibration experiments further show that architecture and augmentation choices have a larger effect on final accuracy than the training-rule modifications studied here. Cumulative free-riding is therefore a real and repairable optimization pathology. Nonetheless, for the FF training rules, architectures, and datasets we study, it is not the dominant factor limiting achievable accuracy.

前向传播神经网络优化特征分离深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。