用信息熵识别冗余块，高效压缩大模型且不损失性能
Entropy-Based Block Pruning for Efficient Large Language Models
- 基于隐藏状态熵识别计算块中的冗余信息
- 在保持精度前提下，模型规模压缩效果优于传统相似度方法
- 适合追求轻量化部署的LLM应用开发者
随着大语言模型持续扩展，其日益增长的计算与存储需求给实际部署带来挑战。本文研究Transformer模型中的冗余问题，提出一种基于信息熵的剪枝策略，在保持性能的同时提升效率。实验表明，隐藏表示的熵在早期块中下降，后续多数块则逐渐上升，说明熵能更准确反映计算块的信息丰富度。相比仅捕捉几何关系的余弦相似度，熵直接衡量不确定性与信息量，是更可靠的剪枝依据。大量实验显示，该方法在减少模型规模的同时维持更高精度，为高效模型部署提供新路径。
原文摘要 · Abstract (English)
As large language models continue to scale, their growing computational and storage demands pose significant challenges for real-world deployment. In this work, we investigate redundancy within Transformer-based models and propose an entropy-based pruning strategy to enhance efficiency while maintaining performance. Empirical analysis reveals that the entropy of hidden representations decreases in the early blocks but progressively increases across most subsequent blocks. This trend suggests that entropy serves as a more effective measure of information richness within computation blocks. Unlike cosine similarity, which primarily captures geometric relationships, entropy directly quantifies uncertainty and information content, making it a more reliable criterion for pruning. Extensive experiments demonstrate that our entropy-based pruning approach surpasses cosine similarity-based methods in reducing model size while preserving accuracy, offering a promising direction for efficient model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。