通过调控权重奇异值减轻激活值异常，提升大模型量化精度
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
- 针对大模型激活值异常问题，仅对最大奇异值对应的权重进行调节
- 在ImageNet上实现W4A4量化下7%的精度提升，结合QAT更达4%增益
- 适用于大规模视觉语言模型，适合追求高效部署的研究者
大规模Transformer模型中的激活值异常是模型量化的主要挑战，导致量化时精度严重下降。我们观察到，随着预训练规模增大（如从CLIP到更充分训练的SigLIP和SigLIP2），异常程度加剧。通过理论分析与实证相关性研究，我们确立了激活异常与权重主导奇异值之间的直接关联。基于此，提出选择性谱衰减（S²D）方法，在微调阶段仅对最大奇异值对应的权重成分进行几何合理正则化。大量实验表明，S²D显著减少激活异常，生成天然适配量化的良好表征。使用S²D训练的模型在ImageNet上实现高达7%的精度提升（W4A4量化），结合量化感知训练（QAT）后进一步提升4%。该效果可推广至下游任务及视觉语言模型，使日益庞大且严格训练的模型仍能保持高效部署能力。
原文摘要 · Abstract (English)
Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay ($S^2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that $S^2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly. Models trained with $S^2D$ achieve up to 7% improved PTQ accuracy on ImageNet under W4A4 quantization and 4% gains when combined with QAT. These improvements also generalize across downstream tasks and vision-language models, enabling the scaling of increasingly large and rigorously trained models without sacrificing deployment efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。