通过块聚类压缩模型内存，64倍无损压缩无需重训练
LegoNet: Memory Footprint Reduction Through Block Weight Clustering
- 将全模型权重按块聚类，不依赖层类型
- ResNet-50在Cifar-10/Imagenet上实现64倍内存压缩无精度损失
- 128倍压缩仅3%精度下降，无需重训练或数据
随着神经网络应用对精度和性能要求不断提升，其规模与内存占用也随之增长。在缓存和内存受限的嵌入式设备上，这种增长限制了其使用前沿模型的能力。本文提出LegoNet，一种压缩技术：不依赖层类型，将整个模型的权重构造成块并进行聚类。采用块而非单个值聚类，使得在不移除任何权重、不改变架构、不重新训练或使用数据的情况下，对在Cifar-10和ImageNet上训练的ResNet-50实现了超过64倍的内存压缩，且精度无损失。同时，通过寻找16个4x4块的排列方式，实现了128倍压缩，精度损失小于3%。所有效果均无需(重)训练或微调。
原文摘要 · Abstract (English)
As the need for neural network-based applications to become more accurate and powerful grows, so too does their size and memory footprint. With embedded devices, whose cache and RAM are limited, this growth hinders their ability to leverage state-of-the-art neural network architectures. In this work, we propose \textbf{LegoNet}, a compression technique that \textbf{constructs blocks of weights of the entire model regardless of layer type} and clusters these induced blocks. Using blocks instead of individual values to cluster the weights, we were able to compress ResNet-50 trained for Cifar-10 and ImageNet with only 32 4x4 blocks, compressing the memory footprint by over a factor of \textbf{64x without having to remove any weights} or changing the architecture and \textbf{no loss to accuracy}, nor retraining or any data, and show how to find an arrangement of 16 4x4 blocks that gives a compression ratio of \textbf{128x with less than 3\% accuracy loss}. This was all achieved with \textbf{no need for (re)training or fine-tuning}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。