arXiv:2605.08137cs.LGcs.AI2026-05中稿 · the 7th Annual Wor…被引 2

压缩大模型会放大偏见,尤其激活感知剪枝最危险。

Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI

论文配图:Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI
图 1 · 摘自论文原文
  • 对比三种剪枝方法在不同稀疏度下的表现,发现激活感知剪枝保留语言能力但偏见加剧。
  • 70%稀疏时,47-59%原无偏见的样本出现新刻板行为,偏见得分飙升83.7%。
  • 真实边缘设备上非结构化剪枝无存储和推理加速优势,不适合物联网部署。

权重剪枝被广泛用于在资源受限的物联网和边缘设备上部署大语言模型,但其对模型公平性的影响仍不明确。我们对三个指令微调模型(Gemma-2-9b-it、Mistral-7B-Instruct-v0.3、Phi-3.5-mini-instruct)在三种剪枝方法(随机、基于大小、Wanda)下,四个稀疏度(10%-70%)上进行了受控实证研究,使用12,148个BBQ偏见基准项与5个随机种子,共生成2,368,860次推理记录。结果揭示‘智能剪枝悖论’:激活感知剪枝(Wanda)几乎完全保留困惑度(如Mistral-7B在50%稀疏度下仅上升3.5%),却导致最大偏见放大,刻板印象依赖分数提升83.7%,70%稀疏度下47-59%原无偏见样本产生新刻板行为。随机剪枝则彻底破坏语言能力(困惑度超$10^4$,达$10^8$),但仅产生随机水平偏见。进一步发现,非结构化剪枝在真实边缘硬件上无法带来存储节省或推理延迟降低,削弱其在物联网部署中的初衷。180组稠密与剪枝模型对比中,141组(78.3%)显著(p<0.05),平均| h | = 0.305。已有量化研究报告最多21%响应在偏见与无偏见状态间切换;本研究显示剪枝导致的切换率高达47-59%,近三倍于量化影响,表明剪枝对对齐性的威胁远高于量化。这些发现说明,仅凭困惑度评估会错误保证行为等价性,边缘部署前必须进行偏见感知验证。

原文摘要 · Abstract (English)

Weight pruning is widely advocated for deploying Large Language Models on resource-constrained IoT and edge devices, yet its impact on model fairness remains poorly understood. We conduct a controlled empirical study of three instruction-tuned models (Gemma-2-9b-it, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct) across three pruning methods (Random, Magnitude, Wanda) at four sparsity levels (10-70%) on 12,148 BBQ bias benchmark items with 5 random seeds, totaling 2,368,860 inference records. Our results reveal a Smart Pruning Paradox: activation-aware pruning (Wanda) preserves perplexity nearly perfectly (just 3.5% increase at 50% sparsity for Mistral-7B), yet produces the highest bias amplification, with Stereotype Reliance Score increasing 83.7% and 47-59% of previously unbiased items developing new stereotypical behaviors at 70% sparsity. Random pruning destroys language capability entirely (perplexity exceeding $10^4$ and reaching $10^8$) but produces only random-chance bias. We further show that unstructured pruning provides zero storage savings and zero inference latency reduction on real edge hardware, undermining the primary motivation for its use in IoT deployment. Of 180 dense-vs-pruned comparisons, 141 (78.3%) are significant ($p < 0.05$) with mean $|h| = 0.305$. Published quantization studies report up to 21% of responses flipping between biased and unbiased states; our pruning results show transition rates nearly three times higher (47-59%), suggesting pruning poses a categorically greater risk to alignment than quantization. These findings demonstrate that perplexity-based evaluation provides false assurance of behavioral equivalence, and that IoT deployment pipelines require bias-aware validation before deploying pruned models at the edge.

大模型压缩偏见放大边缘推理剪枝风险

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。