arXiv:2603.05048cs.LGcs.AR2026-03被引 1

提出新损失函数,让量化神经网络无需训练注入错误就能抗位翻转。

MCEL: Margin-Based Cross-Entropy Loss for Error-Tolerant Quantized Neural Networks

  • 基于输出层分类边界设计新损失,显式增强模型对位错误的容忍度。
  • 在1%错误率下,准确率提升最高达15%,且不依赖训练时注入错误。
  • 参数可解释、易部署,适合未来近似计算和易出错内存场景。

神经网络在新兴近似计算平台和易出错存储技术上的可靠应用,亟需对位错误具备鲁棒性。现有方法通过训练时注入位翻转来实现容错,但存在计算开销大、高错误率下精度下降、难以扩展至大规模模型等问题。本文研究发现,神经网络的位错误容忍能力与输出层分类边界宽度直接相关。基于此,提出一种新的损失函数——边际交叉熵损失(MCEL),在保持标准交叉熵优化特性的同时,显式促进逻辑值层面的边界分离,并引入可解释的边际参数,实现鲁棒性可调。在多个数据集、多种网络结构和量化方案下的大量实验表明,MCEL显著提升位错误容忍度,在1%错误率下准确率最高提升15%。该方法实现简单、效率高,可作为标准交叉熵损失的即插即用替代方案,为近似计算和易出错存储系统提供可扩展、有理论依据的新思路。

原文摘要 · Abstract (English)

Robustness to bit errors is a key requirement for the reliable use of neural networks (NNs) on emerging approximate computing platforms and error-prone memory technologies. A common approach to achieve bit error tolerance in NNs is injecting bit flips during training according to a predefined error model. While effective in certain scenarios, training-time bit flip injection introduces substantial computational overhead, often degrades inference accuracy at high error rates, and scales poorly for larger NN architectures. These limitations make error injection an increasingly impractical solution for ensuring robustness on future approximate computing platforms and error-prone memory technologies. In this work, we investigate the mechanisms that enable NNs to tolerate bit errors without relying on error-aware training. We establish a direct connection between bit error tolerance and classification margins at the output layer. Building on this insight, we propose a novel loss function, the Margin Cross-Entropy Loss (MCEL), which explicitly promotes logit-level margin separation while preserving the favorable optimization properties of the standard cross-entropy loss. Furthermore, MCEL introduces an interpretable margin parameter that allows robustness to be tuned in a principled manner. Extensive experimental evaluations across multiple datasets of varying complexity, diverse NN architectures, and a range of quantization schemes demonstrate that MCEL substantially improves bit error tolerance, up to 15 % in accuracy for an error rate of 1 %. Our proposed MCEL method is simple to implement, efficient, and can be integrated as a drop-in replacement for standard CEL. It provides a scalable and principled alternative to training-time bit flip injection, offering new insights into the origins of NN robustness and enabling more efficient deployment on approximate computing and memory systems.

量化鲁棒性损失函数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。