通过激活方差与稀疏性评估大模型层重要性，可删减四分之一层仍保持90%性能。
AVSS: Layer Importance Evaluation in Large Language Models via Activation Variance-Sparsity Analysis
- 提出AVSS指标，融合激活方差与稀疏性衡量各层贡献。
- 移除最低25%的层后，多项任务性能仍超原模型90%。
- 适合想压缩模型、提升效率的LLM研究者使用。
深度学习中层重要性评估是活跃研究方向，对模型优化与可解释性有重要意义。尽管大语言模型（LLMs）在多个领域广泛应用，但对其内部各层功能重要性及性能贡献的研究仍有限，尤其缺乏从激活分布角度的分析。本文提出激活方差-稀疏性评分（AVSS），通过归一化激活方差与稀疏性结合，量化各层对模型性能的贡献。基于该评分，移除约最低25%的层后，在问答、语言建模和情感分类等任务上仍保持超过90%的原始性能，表明这些层可能非必要。该方法为识别冗余层提供了系统性路径，有助于构建更高效的大型语言模型架构。
原文摘要 · Abstract (English)
The evaluation of layer importance in deep learning has been an active area of research, with significant implications for model optimization and interpretability. Recently, large language models (LLMs) have gained prominence across various domains, yet limited studies have explored the functional importance and performance contributions of individual layers within LLMs, especially from the perspective of activation distribution. In this work, we propose the Activation Variance-Sparsity Score (AVSS), a novel metric combining normalized activation variance and sparsity to assess each layer's contribution to model performance. By identifying and removing approximately the lowest 25% of layers based on AVSS, we achieve over 90% of original model performance across tasks such as question answering, language modeling, and sentiment classification, indicating that these layers may be non-essential. Our approach provides a systematic method for identifying less critical layers, contributing to efficient large language model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。