用低秩结构让局部学习更高效,精度接近反向传播。
Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning
- 在奇异值分解的低秩流形上做局部更新,参数更少。
- 在CIFAR和ImageNet上达到接近反向传播的准确率。
- 适合追求低内存、并行训练的深度学习应用。
深度神经网络通常依赖反向传播(BP)进行训练,虽精度高但需全局误差传播和完整参数化,导致内存与计算开销大。直接反馈对齐(DFA)虽支持局部并行更新且内存更低,但在深层网络尤其是卷积网络中受限于无结构反馈,扩展性差。为此,本文提出一种基于权重矩阵奇异值分解(SVD)定义的低秩流形上的结构化局部学习框架。各层在分解形式下训练,通过融合交叉熵、子空间对齐和正交性正则的复合损失更新SVD分量。反馈矩阵按SVD结构构造,确保前向与反馈路径的一致对齐。该方法在不依赖剪枝或后期压缩的前提下,显著减少可训练参数。在CIFAR-10、CIFAR-100和ImageNet上的实验表明,其精度接近传统反向传播。消融实验证明各损失项在低秩设置下的关键作用。结果确立了在低秩流形上进行局部学习是一种原理清晰且可扩展的全秩梯度训练替代方案。
原文摘要 · Abstract (English)
Training deep neural networks (DNNs) with backpropagation (BP) achieves state-of-the-art accuracy but requires global error propagation and full parameterization, leading to substantial memory and computational overhead. Direct Feedback Alignment (DFA) enables local, parallelizable updates with lower memory requirements but is limited by unstructured feedback and poor scalability in deeper architectures, specially convolutional neural networks. To address these limitations, we propose a structured local learning framework that operates directly on low-rank manifolds defined by the Singular Value Decomposition (SVD) of weight matrices. Each layer is trained in its decomposed form, with updates applied to the SVD components using a composite loss that integrates cross-entropy, subspace alignment, and orthogonality regularization. Feedback matrices are constructed to match the SVD structure, ensuring consistent alignment between forward and feedback pathways. Our method reduces the number of trainable parameters relative to the original DFA model, without relying on pruning or post hoc compression. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method achieves accuracy comparable to that of BP. Ablation studies confirm the importance of each loss term in the low-rank setting. These results establish local learning on low-rank manifolds as a principled and scalable alternative to full-rank gradient-based training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。