实验证明层局部训练在真实数据上难以超越传统反向传播。
Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training
- 提出DTG-FF方法，实现九个真实数据集上的前沿性能
- 224x224图像上仅达49.4%准确率，远低于传统方法的75%以上
- 揭示合成数据高估了层局部训练的可扩展性
前向-前向（FF）学习用严格层内更新替代反向传播。近期研究在32x32合成数据上缩小了与反向传播的差距，引发对真实规模下层局部训练可行性的疑问。为此，我们提出DTG-FF——动态温度评估、解耦归一化与多层融合——在九个真实数据集上达到先进水平（CIFAR-10 91.8%，ImageNet-100 224x224首个FF基线）。实验显示：（1）真实数据下，相同架构和配方下，BP-DeepSup基准在CIFAR-10/CIFAR-100上分别领先DTG-FF 2.40/5.93个百分点，且随类别数增加差距扩大；224x224尺度下，同一方法仅达49.4%，而标准反向传播普遍超75% [Tian et al., 2020]，暴露真实数据上限。 （2）合成与真实任务中存在K冲突：合成任务中随类别数K增加，FF表现持续优于BP，但在真实图像上情况反转且差距扩大。通过在CIFAR-100中区分粗粒度与细粒度标签的对比实验，发现合成任务混淆输出维度与细粒度判别难度，高估了FF迁移能力。 （3）系统审计表明，尽管无需存储深度激活，但受限于8GB硬件，标准BP+梯度累积可实现4.18 GB / 157 img/s，而DTG-FF为7.90 GB / 138 img/s，故以内存优势支持FF在此规模的论断不成立。
原文摘要 · Abstract (English)
Forward-Forward (FF) learning [Hinton, 2022] replaces backpropagation with strictly layer-local goodness updates. Recent FF-CNN work has narrowed the gap to BP on 32x32 benchmarks, raising the question of whether layer-local training is becoming a viable alternative at realistic scale. To probe this rigorously, we develop DTG-FF -- dynamic temperature goodness, decoupled normalization, and multi-layer fusion -- as an instrument that sets FF-family state of the art across nine real-data benchmarks (91.8% CIFAR-10 and the first FF baseline at ImageNet-100 224x224), and use it to audit how far layer-local training actually scales. (1) Real-data scaling. Under identical recipe and backbone, an architecture-matched BP-DeepSup baseline beats DTG-FF by 2.40/5.93 pp on CIFAR-10/CIFAR-100, and the gap widens with class count. At 224x224 the same instrument reaches only 49.4% -- the first FF baseline at this scale, versus typical BP above 75% [Tian et al., 2020] -- exposing a real-data ceiling invisible at 32x32. (2) Synthetic vs. real K-conflict. DTG-FF increasingly outperforms BP as class count K grows on synthetic teacher-student tasks, yet on real images the FF-BP gap reverses sign and widens with K. A within-dataset CIFAR-100 coarse vs. fine probe isolates label-hierarchy from image distribution: synthetic K-sweeps confound output dimensionality with fine-grained discrimination difficulty and thereby overstate FF transferability. (3) Systems audit. FF can be implemented without storing depth-wide activations, but on commodity 8 GB hardware standard BP+gradient-accumulation reaches 4.18 GB / 157 imgs/s versus DTG-FF's 7.90 GB / 138 imgs/s, so a memory-based justification for FF at this scale is not supported under fair baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。