arXiv:2606.03002cs.LGcs.AI2026-06被引 1

量化模型困惑度看似正常,但关键特征可能已受损,需额外检查。

Perplexity Can Miss SAE Feature Damage Under Quantization

论文配图:Perplexity Can Miss SAE Feature Damage Under Quantization
图 1 · 摘自论文原文
  • 用固定稀疏自编码器衡量量化前后特征变化
  • INT7和INT6下分别有18.7%和51.3%特征丢失
  • 特征损失可预测,适合做模型压缩审计

量化是部署大语言模型的常用方法,通常以困惑度或下游准确率接近原模型为判断标准。但行为一致未必意味着特征保真:用于解释全精度模型的稀疏自编码器(SAE)特征在权重量化后可能发生改变。本文通过冻结SAE作为基准,对Pythia-70M和Gemma-2-2B在从INT8到INT4的位宽下,使用相同输入编码全精度与四舍五入量化激活,以皮尔逊相关系数衡量每特征存活率。核心发现:困惑度可能掩盖特征损伤——在Gemma-2-2B上,INT7提升困惑度的同时导致18.7%活跃SAE特征退化;滑窗评估下,INT6也改善困惑度,但仅51.3%活跃特征存活。特征存活呈渐进式而非突变式,Pythia中62.4%、Gemma中51.3%特征在INT6下保留;大部分未存活特征为模糊而非完全失效。特征存活率可仅凭全精度特征统计预测,交叉验证AUC达0.92–0.97,峰值激活为最强单变量预测因子。此外,RTN量化与匹配困惑度的幅度剪枝损伤高度重叠,雅各布重叠0.79–0.86,损伤得分斯皮尔曼相关性达0.98。结果表明,仅依赖行为指标无法保证解释能力迁移,需开展特征级压缩审计。

原文摘要 · Abstract (English)

Quantization is a standard path to deploying large language models, and quantized models are typically judged acceptable when perplexity or downstream accuracy remains close to the full-precision original. But behavioral parity need not imply feature fidelity: the sparse-autoencoder (SAE) features used to interpret a full-precision model may change after weight rounding. We test this directly by using a frozen SAE as a fixed measurement basis, encoding full-precision and round-to-nearest (RTN) quantized activations on identical tokens, and measuring per-feature survival by Pearson correlation across bit-widths from INT8 to INT4 on Pythia-70M and Gemma-2-2B. Our central finding is that perplexity can miss feature damage: on Gemma-2-2B, INT7 improves perplexity while degrading 18.7% of active SAE features, and under sliding-window evaluation INT6 also improves perplexity while only 51.3% of active features survive. Feature survival is graded rather than cliff-like, with 62.4% of active Pythia features and 51.3% of active Gemma features surviving at INT6; most non-surviving features are blurred rather than fully damaged. Survival is also predictable from full-precision feature statistics alone, with cross-validated AUC 0.92--0.97 and peak activation as the strongest marginal predictor. Finally, RTN quantization and matched-perplexity magnitude pruning damage strongly overlapping feature sets, with Jaccard overlap 0.79--0.86 and damage-score Spearman correlation 0.98. These results show that behavioral metrics alone are insufficient evidence that full-precision interpretability findings transfer to quantized models, motivating feature-level audits of compression.

模型量化特征分析可解释性审计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。