提出一种新型量化训练方法，让模型参数自动聚集到离散值。
PARQ: Piecewise-Affine Regularized Quantization
- 用分段线性正则化引导参数向离散值聚类。
- 在视觉任务中达到与现有方法相当的精度。
- 为常用启发式方法提供理论解释，适合模型压缩研究者。
我们提出一种用于大规模机器学习模型量化感知训练（QAT）的原则性方法。具体而言，我们证明凸的分段线性正则化（PAR）能有效促使模型参数向离散值聚类。通过使用聚合近端随机梯度法（AProx）最小化带PAR正则化的损失函数，并证明其具有最后迭代收敛性。该方法将广泛使用的直通估计器（STE）解释为PARQ的渐近形式。实验表明，PARQ在基于卷积和Transformer的视觉任务中取得了具有竞争力的性能。
原文摘要 · Abstract (English)
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models. Specifically, we show that convex, piecewise-affine regularization (PAR) can effectively induce the model parameters to cluster towards discrete values. We minimize PAR-regularized loss functions using an aggregate proximal stochastic gradient method (AProx) and prove that it has last-iterate convergence. Our approach provides an interpretation of the straight-through estimator (STE), a widely used heuristic for QAT, as the asymptotic form of PARQ. We conduct experiments to demonstrate that PARQ obtains competitive performance on convolution- and transformer-based vision tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。