发现剪枝会严重损害关键数据点，影响模型泛化能力。
As easy as PIE: understanding when pruning causes language models to disagree
- 识别出一类易受剪枝伤害的特殊数据点（PIEs）
- PIE数据点占模型泛化能力的关键部分，剪枝后准确率大幅下降
- 长而复杂的文本更易成为PIE，BERT比BiLSTM更敏感
语言模型剪枝通过移除权重、节点等压缩模型，通常只关注整体效率提升而忽略效果损失。研究发现，特定数据子集在剪枝后准确率下降最严重，这类数据点称为PIEs。尽管它们在各类数据集中占比不高，却对模型泛化能力至关重要。在多个NLP数据集和剪枝方法下，无论类别频率如何，剪枝都会显著影响PIEs表现；且BERT比BiLSTM更易受其影响。进一步分析表明，这些高影响数据点具有更长、更复杂的语义结构。该发现揭示了剪枝对模型核心性能的潜在破坏，为理解剪枝机制提供了新视角。代码已公开于https://github.com/pietrotrope/AsEasyAsPIE。
原文摘要 · Abstract (English)
Language Model (LM) pruning compresses the model by removing weights, nodes, or other parts of its architecture. Typically, pruning focuses on the resulting efficiency gains at the cost of effectiveness. However, when looking at how individual data points are affected by pruning, it turns out that a particular subset of data points always bears most of the brunt (in terms of reduced accuracy) when pruning, but this effect goes unnoticed when reporting the mean accuracy of all data points. These data points are called PIEs and have been studied in image processing, but not in NLP. In a study of various NLP datasets, pruning methods, and levels of compression, we find that PIEs impact inference quality considerably, regardless of class frequency, and that BERT is more prone to this than BiLSTM. We also find that PIEs contain a high amount of data points that have the largest influence on how well the model generalises to unseen data. This means that when pruning, with seemingly moderate loss to accuracy across all data points, we in fact hurt tremendously those data points that matter the most. We trace what makes PIEs both hard and impactful to inference to their overall longer and more semantically complex text. These findings are novel and contribute to understanding how LMs are affected by pruning. The code is available at: https://github.com/pietrotrope/AsEasyAsPIE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。