用局部协同优化替代全局反向传播，提升训练效率与鲁棒性
Learning without Global Backpropagation via Synergistic Information Distillation
- 将网络拆分为模块化流水线，每个模块本地优化对真实标签的信念
- 训练速度更快，内存消耗降低，且在噪声标签下表现更稳定
- 无需修改前向推理，可直接替换现有模型中的反向传播
反向传播（BP）虽是深度学习的基础，但存在两大可扩展性瓶颈：更新锁死（模块需等待完整反向传递完成才能更新）和因存储激活值用于梯度计算导致的高内存开销。为解决这些问题，我们提出协同信息蒸馏（SID），一种将深度学习重构为一系列局部协同优化问题的新训练框架。在SID中，深层网络被组织为模块流水线，每个模块被赋予局部目标，以优化对真实标签的概率信念。该目标在贴合真实标签与保持前序模块信念一致性之间取得平衡。通过解耦模块间的反向依赖，SID实现并行训练，消除更新锁死，并显著降低内存需求。同时，该设计保留标准前向推理流程，使SID可作为BP的即插即用替代方案。我们提供了理论基础，证明了SID能保证网络深度增加时性能单调提升。实验表明，SID在分类准确率上持续匹配或超越BP，展现出优越的可扩展性及对标签噪声的强鲁棒性。代码已开源：https://github.com/ychAlbert/sid-bp
原文摘要 · Abstract (English)
Backpropagation (BP), while foundational to deep learning, imposes two critical scalability bottlenecks: update locking, where network modules remain idle until the entire backward pass completes, and high memory consumption due to storing activations for gradient computation. To address these limitations, we introduce Synergistic Information Distillation (SID), a novel training framework that reframes deep learning as a cascade of local cooperative refinement problems. In SID, a deep network is structured as a pipeline of modules, each imposed with a local objective to refine a probabilistic belief about the ground-truth target. This objective balances fidelity to the target with consistency to the belief from its preceding module. By decoupling the backward dependencies between modules, SID enables parallel training and hence eliminates update locking and drastically reduces memory requirements. Meanwhile, this design preserves the standard feed-forward inference pass, making SID a versatile drop-in replacement for BP. We provide a theoretical foundation, proving that SID guarantees monotonic performance improvement with network depth. Empirically, SID consistently matches or surpasses the classification accuracy of BP, exhibiting superior scalability and pronounced robustness to label noise.Code is available at: https://github.com/ychAlbert/sid-bp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。