通过平滑正则化提升模型压缩效果，无需微调即可实现高精度低参数量。
Smooth Model Compression without Fine-Tuning
- 训练时引入核范数与导数惩罚，使权重结构更平滑
- 在CIFAR-10上用70%参数达到91%准确率，无须微调
- 适合追求高效部署的模型压缩场景
大型机器学习模型的压缩与剪枝已成为实际应用部署的关键步骤。传统剪枝和压缩方法通常忽略网络权重的结构特性，限制了其效果。我们研究了平滑正则化对神经网络训练和模型压缩的影响。通过在训练中施加核范数及权重的一阶、二阶导数惩罚，我们鼓励结构化平滑性，同时保持与非平滑模型相当的预测性能。我们发现，标准剪枝方法在这些平滑模型上的表现更优。基于此观察，我们采用基于奇异值分解的压缩方法，利用底层平滑结构，以更小的低秩张量近似原模型权重张量。该方法无需任何微调即可实现当前最优压缩效果：在平滑的ResNet-18上，于CIFAR-10上达到91%准确率，参数量减少70%。
原文摘要 · Abstract (English)
Compressing and pruning large machine learning models has become a critical step towards their deployment in real-world applications. Standard pruning and compression techniques are typically designed without taking the structure of the network's weights into account, limiting their effectiveness. We explore the impact of smooth regularization on neural network training and model compression. By applying nuclear norm, first- and second-order derivative penalties of the weights during training, we encourage structured smoothness while preserving predictive performance on par with non-smooth models. We find that standard pruning methods often perform better when applied to these smooth models. Building on this observation, we apply a Singular-Value-Decomposition-based compression method that exploits the underlying smooth structure and approximates the model's weight tensors by smaller low-rank tensors. Our approach enables state-of-the-art compression without any fine-tuning - reaching up to $91\%$ accuracy on a smooth ResNet-18 on CIFAR-10 with $70\%$ fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。