提出新剪枝方法，让大模型在压缩后更公平、更少偏见。
Debias-SparseGPT: Bias-Aware Pruning for Large Language Models
- 用对比性输入设计二阶项，剪枝时主动消除模型偏见。
- 在25%~50%稀疏度下，比原方法减少偏见，保持性能不变。
- 适合关注模型公平性与部署效率的开发者和研究者。
模型压缩技术如剪枝和量化可提升大语言模型（LLMs）的部署效率。然而，近期研究表明，权重稀疏化方法（如SparseGPT）会放大模型中的固有偏见，导致输出随提示中的人物角色线索显著变化。本文提出Debias-SparseGPT，一种后训练剪枝方法，通过在人口学对比输入上引入二阶项实现表征去偏。我们在多种生成式LLM上进行了实证验证，结果表明：在25%、50%及结构化2:4稀疏度下，Debias-SparseGPT均显著降低剪枝引发的偏见，同时保持模型困惑度与零样本准确率。在最严苛的2:4结构稀疏模式下，通过向校准集添加长上下文、内容丰富的样本，进一步提升了下游任务表现与公平性。总体而言，该方法在保持稀疏模型计算效率的同时，优化了偏见-性能权衡。
原文摘要 · Abstract (English)
Model compression techniques such as pruning and quantization facilitate the efficient deployment and acceleration of Large Language Models (LLMs). However, recent studies show that weight sparsification methods, such as SparseGPT, can amplify existing biases in models, with outputs varying significantly depending on persona cues in the prompt. In this paper, we introduce Debias-SparseGPT, a post-training pruning method incorporating representational debiasing using a second-order term defined over demographically contrasting inputs. We perform empirical validation of our method over a wide range of generative LLMs. Across models and sparsity regimes (25%, 50%, and structured 2:4 sparsity), Debias-SparseGPT consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy. Under the most restrictive 2:4 structured sparsity pattern, which most aggressively degrades model quality, augmenting the calibration set with long-context, content-rich examples further improves both downstream performance and fairness. Overall, Debias-SparseGPT advances the bias-performance trade-off while preserving the computational efficiency of sparse models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。