揭示量化与剪枝的理论关联,证明低精度网络可被精确构造
Quantization vs Pruning: Insights from the Strong Lottery Ticket Hypothesis
- 基于数划分问题,推导量化场景下的随机子集和新理论
- 证明量化网络可被精确表示,且初始网络过参数化程度最优
- 为低精度神经网络设计提供理论依据,适合算法优化研究者
量化是提升神经网络效率的关键技术,但其理论理解仍不充分。已有研究表明,通过剪枝大规模随机初始化网络,可构建极低精度网络(如二值网络),且原网络与剪枝后网络的大小比最多为多项对数级。这一方法启发了强彩票票假设(SLTH)的理论研究,该研究基于随机子集和问题展开。然而,这些结果主要针对连续情形,无法直接推广至量化设置。本文基于Borgs等人关于数划分问题的基础成果,推导出量化场景下随机子集和的新理论结果,并将SLTH框架扩展至有限精度网络。此前工作表明剪枝可近似特定类神经网络,而本文证明在量化设置下,目标离散神经网络可被精确表示,并给出了初始网络过参数化程度与目标网络精度间的最优边界。
原文摘要 · Abstract (English)
Quantization is an essential technique for making neural networks more efficient, yet our theoretical understanding of it remains limited. Previous works demonstrated that extremely low-precision networks, such as binary networks, can be constructed by pruning large, randomly-initialized networks, and showed that the ratio between the size of the original and the pruned networks is at most polylogarithmic. The specific pruning method they employed inspired a line of theoretical work known as the Strong Lottery Ticket Hypothesis (SLTH), which leverages insights from the Random Subset Sum Problem. However, these results primarily address the continuous setting and cannot be applied to extend SLTH results to the quantized setting. In this work, we build on foundational results by Borgs et al. on the Number Partitioning Problem to derive new theoretical results for the Random Subset Sum Problem in a quantized setting. Using these results, we then extend the SLTH framework to finite-precision networks. While prior work on SLTH showed that pruning allows approximation of a certain class of neural networks, we demonstrate that, in the quantized setting, the analogous class of target discrete neural networks can be represented exactly, and we prove optimal bounds on the necessary overparameterization of the initial network as a function of the precision of the target network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。