首次分析1比特量化中直通估计器的样本复杂度,揭示数据量对训练成功的关键作用。
Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization
- 基于双层网络分析直通估计器的收敛性,给出数据维度决定的样本复杂度下限。
- 在标签噪声下发现梯度迭代反复逃离并返回最优二值权重的奇特循环现象。
- 实验证明非高斯数据需归一化才能使直通估计器有效,强调预处理重要性。
训练量化神经网络需应对底层优化问题的非可导与离散特性。为解决此挑战,直通估计器(STE)成为最广泛采用的启发式方法,通过引入有偏但有效的代理梯度,实现离散操作的反向传播。然而其理论性质仍缺乏系统研究,现有分析多假设无限训练数据以考察泛化误差。本文首次对神经网络量化中的STE进行样本复杂度分析。理论结果表明样本数量对STE成功至关重要,这一关键洞察此前未被揭示。具体而言,通过对具有二值权值与激活的双层神经网络进行量化感知训练,我们推导出保证STE优化收敛至全局最小值的样本复杂度边界,涵盖遍历性与非遍历性分析。此外,在标签噪声存在时,证明了STE梯度方法具有独特递归性质:迭代点反复逃离并返回最优二值权值。最后,实验表明STE在一般非高斯数据上失效,但通过归一化可恢复有效性,凸显其在高效量化中的实际意义。
原文摘要 · Abstract (English)
Training quantized neural networks requires addressing the non-differentiable and discrete nature of the underlying optimization problem. To tackle this challenge, the straight-through estimator (STE) has become the most widely adopted heuristic, allowing backpropagation through discrete operations by introducing biased yet valid surrogate gradients. However, its theoretical properties remain largely unexplored, with few existing analyses focus on the generalization error by assuming an infinite amount of training data. In contrast, this work presents the first sample complexity analysis of STE in the context of neural network quantization. Our theoretical results highlight the critical role of sample size in the success of STE, a key insight absent from existing studies. Specifically, by analyzing the quantization-aware training of a two-layer neural network with binary weights and activations, we derive the sample complexity bounds in terms of the data dimensionality that guarantee the convergence of STE-based optimization to the global minimum for both ergodic and non-ergodic analyses. Moreover, in the presence of label noises, we prove an intriguing recurrence property of STE-gradient method, where the iterate repeatedly escape from and return to the optimal binary weights. Finally, we empirically demonstrate that STE fails for general non-Gaussian data but its effectiveness can be restored through normalization, underscoring its practical importance in effective quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。