arXiv:2501.03095cs.CVcs.NE2025-01被引 2

提出无需依赖网络结构的多目标压缩方法,大幅降低模型内存占用。

A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks

  • 用均匀分箱量化权重,构建共享代码本实现高效压缩
  • 通过进化算法搜索最优分箱数,压缩比达14.98倍以上
  • 适用于各类模型和任务,适合嵌入式设备部署

深度神经网络在训练后需存储数百万甚至数十亿个权重,难以部署在资源受限的嵌入式设备上。权重量化共享是一种流行压缩方法,通过减少唯一权重值并共享特定连接中的权重来降低存储开销。本文提出一种不依赖网络架构、维度、任务和数据集的多目标进化算法(MOEA)压缩框架。采用均匀分箱将网络权重量化为单一代码本(查找表),以实现高效表示。利用MOEA优化性能与共享权重数量两个目标,搜索帕累托最优的k个分箱。随后对非支配解集使用迭代合并技术,合并相邻分箱而不降低性能,进一步减少分箱数、提升压缩比。该方法模型与层无关,所有层权重统一聚类;所用均匀量化复杂度为O(N),远低于k-means等非均匀方法的O(Nkt)。同时,直接使用聚类中心作为共享权重,避免昂贵的重训练过程。实验表明,在CIFAR-10上内存可减少13.72~14.98倍,CIFAR-100上减少11.61~12.99倍,ImageNet上减少7.44~8.58倍,验证了该框架的有效性。

原文摘要 · Abstract (English)

Deep neural networks suffer from storing millions and billions of weights in memory post-training, making challenging memory-intensive models to deploy on embedded devices. The weight-sharing technique is one of the popular compression approaches that use fewer weight values and share across specific connections in the network. In this paper, we propose a multi-objective evolutionary algorithm (MOEA) based compression framework independent of neural network architecture, dimension, task, and dataset. We use uniformly sized bins to quantize network weights into a single codebook (lookup table) for efficient weight representation. Using MOEA, we search for Pareto optimal $k$ bins by optimizing two objectives. Then, we apply the iterative merge technique to non-dominated Pareto frontier solutions by combining neighboring bins without degrading performance to decrease the number of bins and increase the compression ratio. Our approach is model- and layer-independent, meaning the weights are mixed in the clusters from any layer, and the uniform quantization method used in this work has $O(N)$ complexity instead of non-uniform quantization methods such as k-means with $O(Nkt)$ complexity. In addition, we use the center of clusters as the shared weight values instead of retraining shared weights, which is computationally expensive. The advantage of using evolutionary multi-objective optimization is that it can obtain non-dominated Pareto frontier solutions with respect to performance and shared weights. The experimental results show that we can reduce the neural network memory by $13.72 \sim14.98 \times$ on CIFAR-10, $11.61 \sim 12.99\times$ on CIFAR-100, and $7.44 \sim 8.58\times$ on ImageNet showcasing the effectiveness of the proposed deep neural network compression framework.

模型压缩权重量化进化算法嵌入式部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。