模型压缩中，表征维度决定剪枝后大模型性能表现。
How can representation dimension dominate structurally pruned LLMs?
- 通过结构化剪枝控制表征维度，研究其对模型影响。
- 表征维度主导线性变换与模型预测结果，决定性能表现。
- 给出性能预测公式，无需实际评估即可估算精度与困惑度。
剪枝假设原始深度神经网络中存在一个子网络，能在计算量更少的情况下达到相近的模型性能。然而，不同子网络提取方式如何影响模型表现尚不明确。本文以表征维度（即嵌入维度、模型维度或残差流维度）为切入点，研究大语言模型（LLM）Transformer模块中的线性变换，并采用特定的结构化剪枝方法SliceGPT提取不同表征维度的子网络。通过机制分析前向传播中的激活流动，发现表征维度主导了线性变换、模型预测乃至最终性能。文中给出了无需实际评估即可计算剪枝后模型性能（困惑度与准确率）的显式解析关系，并在Llama-3-8B-Instruct和Phi-3-mini-4k-Instruct上进行了实证验证。
原文摘要 · Abstract (English)
Pruning assumes a subnetwork exists in the original deep neural network, which can achieve comparative model performance with less computation than the original. However, it is unclear how the model performance varies with the different subnetwork extractions. In this paper, we choose the representation dimension (or embedding dimension, model dimension, the dimension of the residual stream in the relevant literature) as the entry point to this issue. We investigate the linear transformations in the LLM transformer blocks and consider a specific structured pruning approach, SliceGPT, to extract the subnetworks of different representation dimensions. We mechanistically analyse the activation flow during the model forward passes, and find the representation dimension dominates the linear transformations, model predictions, and, finally, the model performance. Explicit analytical relations are given to calculate the pruned model performance (perplexity and accuracy) without actual evaluation, and are empirically validated with Llama-3-8B-Instruct and Phi-3-mini-4k-Instruct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。