揭示剪枝在生成任务中失效的根源，发现概率空间敏感是关键
Demystifying When Pruning Works via Representation Hierarchies
- 从表示层级分析剪枝：分嵌入、logit、概率三阶段
- 概率空间对剪枝扰动最敏感，导致生成时误差累积
- 适合非生成任务剪枝，如检索与选择题，不推荐生成场景
网络剪枝通过移除次要参数或结构来提升效率，但其效果在语言任务中并不稳定：在非生成任务中表现良好，却常在生成任务中失败。为理解这一差异，我们从表示层级视角分析剪枝，将语言模型内部计算分解为三个连续空间：嵌入（隐藏表示）、logit（softmax前输出）和概率（softmax后分布）。研究发现，嵌入与logit空间的表示对剪枝引起的扰动具有较强鲁棒性；但从logit到概率的非线性变换会放大这些偏差，并在生成过程中逐时间步累积，导致性能显著下降。相比之下，分类标记概率子空间的稳定性与嵌入空间的鲁棒性共同支撑了剪枝在检索、多选等非生成任务中的有效性。该分析解耦了剪枝在不同任务中的影响机制，为实际应用提供指导。代码已公开于 https://github.com/CASE-Lab-UMD/Pruning-on-Representations。
原文摘要 · Abstract (English)
Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network pruning from a representation-hierarchy perspective, decomposing the internal computation of language models into three sequential spaces: embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the embedding and logit spaces are largely robust to pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-token probability subspace, together with the robustness of the embedding space, supports the effectiveness of pruning for non-generative tasks such as retrieval and multiple-choice selection. Our analysis disentangles the effects of pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/Pruning-on-Representations
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。