量化压缩让大模型悄悄产生偏见,且现有评估方法根本发现不了。
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

- 在多个模型和精度下系统测试量化对偏见的影响
- 3比特量化使6-21%原本无偏的题目出现新偏见,4比特时已有2.5-5.6%出问题
- 标准指标无法捕捉偏见恶化,适合关注模型公平性的研究者与开发者
大语言模型常通过训练后量化压缩以降低推理成本和内存占用,但该压缩对模型质量的影响仍不明确。现有研究通常仅比较全精度与单一量化版本,依赖聚合偏见指标,并仅评估单一模型族,难以区分渐进退化与阈值相关的安全失效。我们对三个指令微调模型(Qwen2.5-7B、Mistral-7B、Phi-3.5-mini)在五种精度级别(BF16至3比特)下,于12,148个BBQ偏见基准样本上进行受控实证研究,共生成911,100条推理记录,覆盖5个随机种子。结果表明,3比特量化使6-21%原本无偏的样本产生新的刻板行为,呈明确剂量-反应关系,经逻辑回归验证;同时模型选择“未知”答案的意愿下降17.4%。关键的是,这些逐项变化被标准质量指标完全掩盖:8比特时困惑度上升不足0.5%,4比特时低于3%,但已有2.5%-5.6%的样本出现新偏见。这说明聚合评估指标会系统性遗漏关键公平性退化,凸显部署前需引入显式偏见检测的质量感知压缩协议的必要性。
原文摘要 · Abstract (English)
Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment, yet the impact of this compression on model quality remains poorly understood. Existing studies typically compare only two conditions (full-precision vs. a single quantized variant), rely on aggregate bias metrics, and evaluate a single model family, making it impossible to distinguish gradual degradation from threshold-dependent safety failures. We conduct a controlled empirical study of three instruction-tuned models (Qwen2.5-7B, Mistral-7B, Phi-3.5-mini) at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items across 5 random seeds, totaling 911,100 inference records. Our results reveal that 3-bit quantization causes 6-21% of previously unbiased items to develop new stereotypical behaviors, following a clear dose-response pattern confirmed via logistic regression, while models' willingness to select "unknown" answers declines by 17.4%. Crucially, these item-level changes are invisible to standard quality metrics: perplexity increases by less than 0.5% at 8-bit and under 3% at 4-bit across all three models, yet 2.5-5.6% of items already develop new biases at 4-bit. These findings demonstrate that aggregate evaluation metrics systematically miss fairness-critical degradation, underscoring the need for quality-aware compression protocols that explicitly test for bias emergence before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。