训练抗权重扰动的神经网络，提升模型在低功耗硬件上的稳定性。
Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbations
- 用RWP和SAM方法寻找对权重扰动鲁棒的极小值点。
- 小噪声下SAM表现更优，大噪声时因梯度消失导致性能下降。
- 动态调整扰动强度可优化训练过程，适合边缘计算场景。
神经网络损失曲面中的平坦区域长期被认为与更好的泛化能力相关。本文研究如何训练对内部权重扰动具有鲁棒性的模型，这对未来低功耗硬件平台尤为重要。我们探索了两种方法：尖锐感知最小化（SAM）和随机权重扰动（RWP），以应对多种权重随机损坏情况。从泛化和优化两个角度出发，我们理论上和实证上证明：过度正则化的RWP训练目标在抗噪声泛化方面最优。对于小幅度噪声，SAM的对抗性目标优于任何RWP配置；但在大噪声下表现不佳，原因在于损失曲面不均匀引发的梯度消失效应，影响了SAM和RWP。最后，我们展示了动态调整扰动强度以匹配损失曲面演化，能有效提升对扰动目标的优化性能。
原文摘要 · Abstract (English)
Flat regions of the neural network loss landscape have long been hypothesized to correlate with better generalization properties. A closely related but distinct problem is training models that are robust to internal perturbations to their weights, which may be an important need for future low-power hardware platforms. In this paper, we explore the usage of two methods, sharpness-aware minimization (SAM) and random-weight perturbation (RWP), to find minima robust to a variety of random corruptions to weights. We consider the problem from two angles: generalization (how do we reduce the noise-robust generalization gap) and optimization (how do we maximize performance from optimizers when subject to strong perturbations). First, we establish, both theoretically and empirically, that an over-regularized RWP training objective is optimal for noise-robust generalization. For small-magnitude noise, we find that SAM's adversarial objective further improves performance over any RWP configuration, but performs poorly for large-magnitude noise. We link the cause of this to a vanishing-gradient effect, caused by unevenness in the loss landscape, affecting both SAM and RWP. Lastly, we demonstrate that dynamically adjusting the perturbation strength to match the evolution of the loss landscape improves optimizing for these perturbed objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。