用谱熵自适应选择压缩每层神经网络权重的秩，提升压缩效率。
Low-Rank Matrix Approximation for Neural Network Compression
- 基于奇异值谱熵动态决定每层权重矩阵的压缩秩
- 相比固定秩压缩，相同压缩率下模型精度更高
- 适合需要高效部署的边缘设备或移动端模型压缩
深度神经网络因庞大的内存和计算需求面临部署挑战。本文提出一种新的自适应秩奇异值分解（ARSVD）方法，通过谱熵来近似最优压缩秩。与传统SVD方法在所有层使用固定秩截断不同，ARSVD根据各层奇异值分布自适应选择压缩秩，确保每层保留一定信息量，减少冗余。该方法实现分层高效压缩，在降低空间和时间复杂度的同时，优于静态秩压缩技术的性能表现。
原文摘要 · Abstract (English)
Deep Neural Networks (DNNs) have encountered an emerging deployment challenge due to large and expensive memory and computation requirements. In this paper, we present a new Adaptive-Rank Singular Value Decomposition (ARSVD) method that approximates the optimal rank for compressing weight matrices in neural networks using spectral entropy. Unlike conventional SVD-based methods that apply a fixed-rank truncation across all layers, ARSVD uses an adaptive selection of the rank per layer through the entropy distribution of its singular values. This approach ensures that each layer will retain a certain amount of its informational content, thereby reducing redundancy. Our method enables efficient, layer-wise compression, yielding improved performance with reduced space and time complexity compared to static-rank reduction techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。