通过噪声注入提升大模型量化稳定性，缓解注意力机制对精度损失的敏感性。
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
- 基于雅可比范数动态调节噪声方差，精准控制注意力层敏感度。
- 在低比特量化下，图像任务准确率提升37%，语言模型困惑度降低40%。
- 适用于追求高鲁棒性量化的研究人员，尤其适合部署在资源受限设备上。
大型语言模型（LLM）的量化常因自注意力机制对离散化误差敏感而受阻。我们发现，软最大函数是量化稳定性的瓶颈，因其对异常值和状态依赖的雅可比矩阵敏感。理论上证明，抑制该雅可比范数有助于约束量化引起的性能下降。基于此，提出雅可比引导噪声注入训练策略：向注意力前的logits注入均值为零、方差由雅可比Frobenius范数决定的高斯噪声。相比以往依赖启发式或直接惩罚雅可比的方法，本方法能根据局部注意力敏感度自动确定最优噪声方差。在多个SOTA LLM架构上评估表明，该方法在主流量化方法之上显著提升鲁棒性。实证分析显示，在低比特量化设置下，SigLIP在ImageNet-1K上的Top-1准确率相对提升最高达37%，语言模型在WikiText上的困惑度相对改善最高达40%。
原文摘要 · Abstract (English)
Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bottleneck for quantization stability due to its sensitivity to outliers and state-dependent Jacobian. We theoretically establish that suppressing the norm of this Jacobian helps in bounding quantization-induced performance degradation. Based on this, we propose Jacobian-Guided Noise Injection, a training strategy that injects zero-mean Gaussian noise into pre-attention logits, with variance derived directly from the Jacobian Frobenius norm. Unlike prior approaches that rely on heuristic or penalise jacobian directly, our method provides a way to identify the optimal noise variance based on the local attention sensitivity. We evaluate the method on SOTA LLM architectures, where it demonstrates improved robustness over popular PTQ methods. Empirical analysis reveals that the proposed method gives up to +37% relative gains on Top-1 accuracy on ImageNet-1K for SigLIP and improves relative perplexity by upto 40% on WikiText for language models in low bit quantisation settings, proving the efficacy of the approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。