提出自适应梯度丢弃法，有效提升深度网络泛化能力
Provable Generalization Bounds for Deep Neural Networks with Momentum-Adaptive Gradient Dropout
- 根据梯度与动量动态调整激活值丢弃率
- 理论证明可使泛化界比传统方法紧29.2%
- 在MNIST和CIFAR-10上泛化误差极小，适合高风险场景
深度神经网络虽性能优异，但因容量过高常过拟合。本文提出动量自适应梯度丢弃（MAGDrop），通过当前梯度与累积动量动态调节激活值丢弃率，增强非凸优化下的稳定性。我们建立了非渐近、可计算的PAC-Bayes泛化界，考虑其自适应特性，利用动量驱动的扰动控制，使边界相比标准方法最紧达29.2%。实验表明，基于激活的MAGDrop在MNIST上达到99.52%准确率，泛化差距仅0.48%；在CIFAR-10上准确率为92.03%，泛化差距为6.52%。提供完整可复现代码与边界数值计算，验证理论结论。本工作连接理论与实践，为提升DNN泛化能力提供稳健框架，适用于高风险应用场景。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) achieve remarkable performance but often suffer from overfitting due to their high capacity. We introduce Momentum-Adaptive Gradient Dropout (MAGDrop), a novel regularization method that dynamically adjusts dropout rates on activations based on current gradients and accumulated momentum, enhancing stability in non-convex optimization landscapes. To theoretically justify MAGDrop's effectiveness, we derive a non-asymptotic, computable PAC-Bayes generalization bound that accounts for its adaptive nature, achieving up to 29.2\% tighter bounds compared to standard approaches by leveraging momentum-driven perturbation control. Empirically, the activation-based MAGDrop achieves competitive performance on MNIST (99.52\%) and CIFAR-10 (92.03\%), with generalization gaps of 0.48\% and 6.52\%, respectively. We provide fully reproducible code and numerical computation of our bounds to validate our theoretical claims. Our work bridges theoretical insights and practical advancements, offering a robust framework for enhancing DNN generalization, making it suitable for high-stakes applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。