通过权重耦合实现模型压缩，支持混合精度量化。
Soft Quantization: Model Compression Via Weight Coupling
- 训练时引入权重短程吸引力，诱导权重离散化。
- 在ResNet-20/CIFAR-10上优于直方图均衡的后训练量化。
- 仅需两个超参数，适合研究压缩与泛化权衡。
我们证明，在训练过程中为神经网络权重引入短程吸引力耦合，可开辟一种新型模型量化路径。该耦合能快速促使模型权重分布离散化，且在仅依赖两个额外超参数的前提下实现混合精度量化。在合适的超参数范围内，我们的‘软量化’方案在ResNet-20/CIFAR-10上的表现优于直方图均衡的后训练量化。软量化不仅提供了一种灵活的机器学习模型压缩新流程，也为研究高维损失景观中压缩与泛化之间的权衡关系提供了新工具。
原文摘要 · Abstract (English)
We show that introducing short-range attractive couplings between the weights of a neural network during training provides a novel avenue for model quantization. These couplings rapidly induce the discretization of a model's weight distribution, and they do so in a mixed-precision manner despite only relying on two additional hyperparameters. We demonstrate that, within an appropriate range of hyperparameters, our "soft quantization'' scheme outperforms histogram-equalized post-training quantization on ResNet-20/CIFAR-10. Soft quantization provides both a new pipeline for the flexible compression of machine learning models and a new tool for investigating the trade-off between compression and generalization in high-dimensional loss landscapes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。