量化训练让神经网络更抗硬件噪声,实测效果优于直接加噪训练。
On Hardening DNNs against Noisy Computations
- 用固定缩放因子的量化感知训练提升模型抗噪能力
- 在多种网络结构上,量化训练使精度下降低于5%
- 加噪训练更优,尤其适合复杂架构的鲁棒性需求
深度学习的成功推动了针对神经网络推理高算力需求的专用硬件设计。随着数字CMOS处理器进一步微型化变得愈发困难,模拟计算等替代范式逐渐受到关注。特别是在矩阵乘法这类计算密集型任务中,模拟计算因能显著提升能效而具有潜力。然而,模拟计算固有的噪声使得维持深度神经网络的高精度变得困难。本文研究了通过量化训练增强神经网络对噪声的鲁棒性。实验结果表明,采用固定缩放因子的量化感知训练可有效提升抗噪能力。我们将其与注入噪声的训练方法(模拟推理时的噪声)进行比较,两者均提升了对噪声的容忍度,但在复杂神经网络架构中,加噪训练表现更优,能实现更强的鲁棒性。
原文摘要 · Abstract (English)
The success of deep learning has sparked significant interest in designing computer hardware optimized for the high computational demands of neural network inference. As further miniaturization of digital CMOS processors becomes increasingly challenging, alternative computing paradigms, such as analog computing, are gaining consideration. Particularly for compute-intensive tasks such as matrix multiplication, analog computing presents a promising alternative due to its potential for significantly higher energy efficiency compared to conventional digital technology. However, analog computations are inherently noisy, which makes it challenging to maintain high accuracy on deep neural networks. This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise. Experimental results across various network architectures show that quantization-aware training with constant scaling factors enhances robustness. We compare these methods with noisy training, which incorporates a noise injection during training that mimics the noise encountered during inference. While both two methods increase tolerance against noise, noisy training emerges as the superior approach for achieving robust neural network performance, especially in complex neural architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。