提出新型剪枝方法，让压缩后的大模型总结观点更公平
Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions
- 基于梯度与激活值设计新剪枝策略，识别冗余但影响输出的参数
- 在多个模型和任务上，剪枝后公平性不降反升，优于现有方法
- 适合关注模型压缩中公平性问题的研究者和应用开发者
通过训练后剪枝实现模型压缩，可在不显著影响性能的前提下减小模型规模与计算开销。然而，剪枝对大语言模型生成观点摘要时公平性的影响尚未被研究，尤其在可能影响公众认知的场景中。本文对三种先进剪枝方法及多种校准集，在三个开源大模型上，使用四种公平性指标进行了系统性实证分析。结果表明，剪枝方法对公平性的影响力大于校准集。基于此，我们提出高梯度低激活（HGLA）剪枝方法，可识别并移除对输入处理冗余但对输出生成影响大的参数。实验显示，HGLA在保持甚至提升公平性方面优于现有方法，且在多模型、多任务中表现稳定。人工评估进一步验证其生成内容更公平。代码已公开：https://github.com/amberhuang01/HGLA。
原文摘要 · Abstract (English)
Model compression through post-training pruning offers a way to reduce model size and computational requirements without significantly impacting model performance. However, the effect of pruning on the fairness of LLM-generated summaries remains unexplored, particularly for opinion summarisation where biased outputs could influence public views.In this paper, we present a comprehensive empirical analysis of opinion summarisation, examining three state-of-the-art pruning methods and various calibration sets across three open-source LLMs using four fairness metrics. Our systematic analysis reveals that pruning methods have a greater impact on fairness than calibration sets. Building on these insights, we propose High Gradient Low Activation (HGLA) pruning, which identifies and removes parameters that are redundant for input processing but influential in output generation. Our experiments demonstrate that HGLA can better maintain or even improve fairness compared to existing methods, showing promise across models and tasks where traditional methods have limitations. Our human evaluation shows HGLA-generated outputs are fairer than existing state-of-the-art pruning methods. Code is available at: https://github.com/amberhuang01/HGLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。