压缩大模型会不均等地损害知识保留，且隐藏真实偏差风险。
The Asymmetric Harms of LLM Compression
- 对比头尾知识，压缩更严重损失头部知识
- 模型对丢失知识仍保持高自信错误回答
- 整体偏差评分稳定，但群体间偏好反转
大型语言模型（LLMs）压缩可降低部署成本，但标准的综合指标如困惑度和准确率常掩盖潜在行为变化。本文系统评估了3个LLM在11种压缩方法下的表现，研究压缩对知识保留、模型置信度及社会偏见的影响。结果发现，压缩显著降低头部知识的相对保留率，而尾部知识受影响较小。此外，压缩后的模型在新丢失的知识上仍表现出高度自信的错误判断。最后，我们发现稳定的整体偏差分数可能掩盖不同人口子群体间相反的刻板印象偏好变化。这些结果揭示了聚合性能指标无法捕捉的不对称行为改变，强调在部署前需对压缩模型进行细粒度评估。
原文摘要 · Abstract (English)
Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias. We find that compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge. Furthermore, compressed models often remain substantially confident in their incorrect answers on newly lost knowledge. Finally, we demonstrate that stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Together, these findings reveal asymmetric behavioral changes that aggregate performance measures fail to capture, highlighting the need for granular evaluation of compressed models before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。