让深度模型适应模拟计算中的噪声变化,提升稳定性。
Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations
- 训练时动态模拟噪声变化,增强模型抗干扰能力
- 在CIFAR-10上准确率从79.3%提升至97.6%
- 适合部署于模拟计算芯片的AI模型优化
深度学习的计算需求与硬件能力之间的差距正急剧扩大。尽管深度学习在众多任务中表现卓越,但其对算力和能耗的要求已超出专用神经处理单元(如Apple Neural Engine、NVIDIA TensorCores)的可持续极限。这一挑战因CMOS工艺缩放放缓而加剧。模拟计算通过直接操控电流、电压、电荷或光子等物理量,可显著提升能效,但其固有缺陷——制造差异、非线性与噪声——导致预测精度下降。现有最有效的方法是噪声训练(Noisy Training),即在训练阶段引入噪声以增强模型对推理阶段扰动的鲁棒性。然而,在真实环境中,噪声特性受温度波动和时间漂移等因素影响而动态变化,导致该方法性能退化。本文强调噪声训练的必要性,同时揭示其在动态噪声下的根本局限。为此,我们提出方差感知噪声训练(Variance-Aware Noisy Training),通过引入模拟推理阶段演化噪声的噪声调度策略,显著提升模型鲁棒性,且无额外训练开销。实验表明,在CIFAR-10上,准确率从79.3%提升至97.6%;在Tiny ImageNet上,从32.4%提升至99.7%。
原文摘要 · Abstract (English)
The disparity between the computational demands of deep learning and the capabilities of compute hardware is expanding drastically. Although deep learning achieves remarkable performance in countless tasks, its escalating requirements for computational power and energy consumption surpass the sustainable limits of even specialized neural processing units, including the Apple Neural Engine and NVIDIA TensorCores. This challenge is intensified by the slowdown in CMOS scaling. Analog computing presents a promising alternative, offering substantial improvements in energy efficiency by directly manipulating physical quantities such as current, voltage, charge, or photons. However, it is inherently vulnerable to manufacturing variations, nonlinearities, and noise, leading to degraded prediction accuracy. One of the most effective techniques for enhancing robustness, Noisy Training, introduces noise during the training phase to reinforce the model against disturbances encountered during inference. Although highly effective, its performance degrades in real-world environments where noise characteristics fluctuate due to external factors such as temperature variations and temporal drift. This study underscores the necessity of Noisy Training while revealing its fundamental limitations in the presence of dynamic noise. To address these challenges, we propose Variance-Aware Noisy Training, a novel approach that mitigates performance degradation by incorporating noise schedules which emulate the evolving noise conditions encountered during inference. Our method substantially improves model robustness, without training overhead. We demonstrate a significant increase in robustness, from 79.3\% with conventional Noisy Training to 97.6\% with Variance-Aware Noisy Training on CIFAR-10 and from 32.4\% to 99.7\% on Tiny ImageNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。