arXiv:2510.16088cs.CVcs.LG2025-10

提出可微分的位移量化方法,训练15轮即可接近全精度性能。

Differentiable, Bit-shifting, and Scalable Quantization without training neural network from scratch

  • 采用可微分位移量化,支持任意比特数,无需手动调导数。
  • 仅用15轮训练,权重量化后在ImageNet上损失低于1%准确率。
  • 适合部署到低算力设备,无需高精度乘法运算,兼容主流模型。

神经网络量化可降低推理时的计算与内存开销。现有方法普遍采用非可微量化,反向传播中梯度需人工设定,导致学习能力存疑。本文提出一种可微分、可扩展的位移量化方法,并证明其收敛至最优网络。相比以往仅对权重进行量化或精度不足的方法,本方法同时支持权重与激活的量化,且能实现多比特(如n比特)量化。在ImageNet数据集上使用ResNet18进行图像分类任务测试,仅权重量化即达到全精度损失小于1%的性能,仅需15轮训练;在权重与激活联合量化下,性能接近当前最先进方法,推理成本仅略高于1比特量化(无对数量化),且不依赖高精度乘法运算。

原文摘要 · Abstract (English)

Quantization of neural networks provides benefits of inference in less compute and memory requirements. Previous work in quantization lack two important aspects which this work provides. First almost all previous work in quantization used a non-differentiable approach and for learning; the derivative is usually set manually in backpropogation which make the learning ability of algorithm questionable, our approach is not just differentiable, we also provide proof of convergence of our approach to the optimal neural network. Second previous work in shift/logrithmic quantization either have avoided activation quantization along with weight quantization or achieved less accuracy. Learning logrithmic quantize values of form $2^n$ requires the quantization function can scale to more than 1 bit quantization which is another benifit of our quantization that it provides $n$ bits quantization as well. Our approach when tested with image classification task using imagenet dataset, resnet18 and weight quantization only achieves less than 1 percent accuracy compared to full precision accuracy while taking only 15 epochs to train using shift bit quantization and achieves comparable to SOTA approaches accuracy in both weight and activation quantization using shift bit quantization in 15 training epochs with slightly higher(only higher cpu instructions) inference cost compared to 1 bit quantization(without logrithmic quantization) and not requiring any higher precision multiplication.

量化可微分位移量化低比特

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。