通过主特征向量投影梯度，提升资源受限下的训练效率
Dynamic Spectral Backpropagation for Efficient Neural Network Training
- 将梯度投影到主特征向量上，降低计算复杂度
- 在多个数据集上优于SAM、LoRA和MAML等方法
- 适合硬件受限场景下的高效训练与少样本学习
动态谱反向传播（DSBP）通过将梯度投影至主特征向量，在资源受限条件下提升神经网络训练效率，并促进平坦极小值的搜索。提出五种扩展：动态谱推理、谱架构优化、谱元学习、谱迁移正则化及李代数启发的动力学机制，以应对鲁棒性、少样本学习和硬件效率挑战。基于三阶随机微分方程（SDE）与PAC Bayes界支撑，实验表明其在CIFAR-10、Fashion MNIST、MedMNIST和Tiny ImageNet上优于尖锐度感知最小化（SAM）、低秩适配（LoRA）和模型无关元学习（MAML）。未来工作聚焦可扩展性、偏见缓解与伦理考量。
原文摘要 · Abstract (English)
Dynamic Spectral Backpropagation (DSBP) enhances neural network training under resource constraints by projecting gradients onto principal eigenvectors, reducing complexity and promoting flat minima. Five extensions are proposed, dynamic spectral inference, spectral architecture optimization, spectral meta learning, spectral transfer regularization, and Lie algebra inspired dynamics, to address challenges in robustness, fewshot learning, and hardware efficiency. Supported by a third order stochastic differential equation (SDE) and a PAC Bayes limit, DSBP outperforms Sharpness Aware Minimization (SAM), Low Rank Adaptation (LoRA), and Model Agnostic Meta Learning (MAML) on CIFAR 10, Fashion MNIST, MedMNIST, and Tiny ImageNet, as demonstrated through extensive experiments and visualizations. Future work focuses on scalability, bias mitigation, and ethical considerations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。