利用神经元线性特性实现模型无损压缩，可减至原大小1/4。
Linearity-based neural network compression
- 基于激活函数特性识别可线性合并的层，实现权重压缩。
- 多数模型压缩至原始大小的1/4且无性能损失。
- 可与剪枝等技术结合，适合部署优化场景。
在神经网络压缩中，现有方法主要通过度量参数重要性和冗余来减少冗余参数。为补充已高度优化的现有方案，我们提出一种基于线性的新型压缩方法。该方法基于直观假设：使用ReLU类激活函数时，始终被激活的神经元行为近似线性，从而允许合并后续层。我们建立了该压缩方法的理论基础，并进行了实验评估。结果表明，该方法在多数测试模型上实现了无损压缩至原始模型大小的1/4。将该方法应用于已通过重要性剪枝的模型时，不同压缩方式之间干扰极小，证明了多种技术组合的可行性。总体而言，本工作为新型压缩方法奠定了基础，有助于构建更小、更高效的神经网络模型。
原文摘要 · Abstract (English)
In neural network compression, most current methods reduce unnecessary parameters by measuring importance and redundancy. To augment already highly optimized existing solutions, we propose linearity-based compression as a novel way to reduce weights in a neural network. It is based on the intuition that with ReLU-like activation functions, neurons that are almost always activated behave linearly, allowing for merging of subsequent layers. We introduce the theory underlying this compression and evaluate our approach experimentally. Our novel method achieves a lossless compression down to 1/4 of the original model size in over the majority of tested models. Applying our method on already importance-based pruned models shows very little interference between different types of compression, demonstrating the option of successful combination of techniques. Overall, our work lays the foundation for a new type of compression method that enables smaller and ultimately more efficient neural network models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。