用互信息优化大模型剪枝，更稳定高效地压缩模型
MI-PRUN: Optimize Large Language Model Pruning via Mutual Information
- 通过隐藏状态变化评估块间互信息，识别冗余模块
- 结合数据处理不等式，揭示整块与单块重要性的关系
- 提出快速选块算法，实现全局最优且效率显著提升
大型语言模型在各领域广泛应用，但需消耗大量计算与内存资源。模型剪枝通过移除冗余组件缓解此问题，其中块剪枝可实现显著压缩与推理加速。然而，现有块剪枝方法常不稳定，难以达到全局最优。本文提出基于互信息的剪枝方法MI-PRUN，利用互信息衡量隐藏状态转移，识别冗余块；结合数据处理不等式（DPI），揭示连续整块与单个块重要性的关系；并设计快速选块算法（Fast-Block-Select），迭代优化块组合，在保证全局最优的同时大幅提升效率。在多种模型和数据集上的实验表明，该方法具有稳定性与有效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become indispensable across various domains, but this comes at the cost of substantial computational and memory resources. Model pruning addresses this by removing redundant components from models. In particular, block pruning can achieve significant compression and inference acceleration. However, existing block pruning methods are often unstable and struggle to attain globally optimal solutions. In this paper, we propose a mutual information based pruning method MI-PRUN for LLMs. Specifically, we leverages mutual information to identify redundant blocks by evaluating transitions in hidden states. Additionally, we incorporate the Data Processing Inequality (DPI) to reveal the relationship between the importance of entire contiguous blocks and that of individual blocks. Moreover, we develop the Fast-Block-Select algorithm, which iteratively updates block combinations to achieve a globally optimal solution while significantly improving the efficiency. Extensive experiments across various models and datasets demonstrate the stability and effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。