用李雅普诺夫函数控制神经网络状态,解决低精度硬件中的数值溢出问题。
Lyapunov-Guided Training for Hardware-Safe Neural Networks Under Fixed-Point Arithmetic
- 通过分层李雅普诺夫函数监控隐藏状态能量,施加单调投影约束
- 12位下实现86.55%准确率,溢出率降至0.012%以下
- 适合部署在资源受限硬件的低精度神经网络训练与推理
低精度神经网络在资源受限硬件中具有吸引力,但定点运算会引入理想量化模型难以捕捉的故障模式。特别是反码溢出会导致隐藏激活值的大小和符号同时改变,引发不稳定的数值误差传播并造成严重精度下降。本文提出一种基于李雅普诺夫稳定的低精度神经网络量化框架,适用于硬件风格的环绕式定点运算。通过分层李雅普诺夫函数监测隐藏状态能量,并施加单调投影以保证状态演化在深度方向上保持有界且非递增。在MNIST数据集上,使用紧凑的基于块的Transformer进行后训练量化与量化感知训练,固定点位宽为4至16位。蒙特卡洛实验显示,无约束的环绕式量化感知训练在6-16位下崩溃至接近随机水平,激活溢出率超过11%。相比之下,所提单调李雅普诺夫投影将激活溢出抑制在0.012%以下,恢复了稳定低精度学习,在12位时达到86.55%准确率。结果表明,基于李雅普诺夫的状态控制可作为可靠的硬件感知稳定机制,用于固定点神经网络的推断与训练。
原文摘要 · Abstract (English)
Low-precision neural networks are attractive for resource-constrained hardware, but fixed-point arithmetic introduces failure modes that are often hidden by idealised quantisation models. In particular, two's-complement overflow wrapping can corrupt hidden activations by changing both their magnitude and sign, leading to unstable numerical error propagation and severe accuracy degradation. This paper proposes a Lyapunov-stabilised quantisation framework for low-precision neural networks operating under hardware-style wrapping arithmetic. The hidden-state energy is monitored through a layerwise Lyapunov function, and a monotone projection is applied to enforce bounded and non-increasing state evolution across depth. The method is evaluated on MNIST using a compact patch-based transformer under post-training quantisation and quantisation-aware training with fixed-point bit-widths from 4 to 16 bits. Monte Carlo results show that unconstrained wrapped quantisation-aware training collapses to near-chance accuracy across 6-16 bits, with activation overflow rates exceeding 11%. In contrast, the proposed monotone Lyapunov projection suppresses activation overflow to below 0.012% and restores stable low-precision learning, achieving 86.55% accuracy at 12 bits. These results demonstrate that Lyapunov-based state control can act as a hardware-aware stabilisation mechanism for reliable fixed-point neural inference and training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。