提出神经语义归因，提升大模型剪枝的可解释性与性能。
Revisiting Large Language Model Pruning using Neuron Semantic Attribution
- 基于神经语义归因，为剪枝后的神经元赋予语义标签。
- 在24个数据集上发现剪枝方法在情感分类任务中性能显著下降。
- 强调校准集选择对剪枝效果的影响，适合模型压缩研究者。
模型剪枝技术对于通过减小模型规模和计算开销来加速大语言模型至关重要。然而，现有剪枝方法在不同数据集和任务上的泛化能力尚不明确。因此，我们使用主流剪枝方法，在24个数据集和4项任务上进行了广泛评估。基于这些评估，我们发现校准集的选择显著影响剪枝方法的性能。此外，我们意外发现现有剪枝方法在情感分类任务中出现显著性能下降。为理解性能下降与被剪枝神经元之间的关系，我们提出神经语义归因（Neuron Semantic Attribution），该方法学习将每个神经元与特定语义相关联。此方法首先使大语言模型中未被剪枝的神经元具备可解释性。
原文摘要 · Abstract (English)
Model pruning technique is vital for accelerating large language models by reducing their size and computational requirements. However, the generalizability of existing pruning methods across diverse datasets and tasks remains unclear. Thus, we conduct extensive evaluations on 24 datasets and 4 tasks using popular pruning methods. Based on these evaluations, we find and then investigate that calibration set greatly affect the performance of pruning methods. In addition, we surprisingly find a significant performance drop of existing pruning methods in sentiment classification tasks. To understand the link between performance drop and pruned neurons, we propose Neuron Semantic Attribution, which learns to associate each neuron with specific semantics. This method first makes the unpruned neurons of LLMs explainable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。