提出新方法让神经网络在任意精度和稀疏度下稳定训练,突破低比特瓶颈。
Robust Training of Neural Networks at Arbitrary Precision and Sparsity
- 将量化视为加性噪声,构建完整可微的前向后向路径
- 实现A1W1和亚1比特模型稳定训练,性能优于现有方法
- 适用于大模型高效部署,为超低资源推理提供理论支持
量化和稀疏化固有的不连续操作长期阻碍反向传播,尤其在极低精度和稀疏场景。尽管社区普遍认为量化不友好于梯度下降,我们首次指出根本问题在于缺乏学习量化噪声鲁棒性的有效梯度路径。标准直通估计器(STE)因前向感知量化而反向忽略,导致误差累积与不稳定。我们通过将量化显式建模为加性噪声,建立完整的前向-后向路径,无需启发式梯度估计。作为自然解法,我们引入基于岭回归的去噪解量化变换,构造明确的修正梯度路径,使训练对噪声具有鲁棒性。我们将该思想扩展至稀疏化,将其视为零化小值的特殊量化形式。统一框架支持任意精度与稀疏度下的模型训练,采用通用训练方案,实现稳定A1W1和亚1比特网络,显著优于现有方法,在现代大模型中开辟效率新边界,为超高效神经网络提供理论基础。
原文摘要 · Abstract (English)
The discontinuous operations inherent in quantization and sparsification introduce a long-standing obstacle to backpropagation, particularly in ultra-low precision and sparse regimes. While the community has long viewed quantization as unfriendly to gradient descent due to its lack of smoothness, we pinpoint-for the first time-that the key issue is the absence of a proper gradient path that allows training to learn robustness to quantization noise. The standard Straight-Through Estimator (STE) exacerbates this with its well-understood mismatch: a quantization-aware forward pass but oblivious backward pass, leading to unmanaged error and instability. We solve this by explicitly modeling quantization as additive noise, making the full forward-backward path well-defined without heuristic gradient estimation. As one natural solution, we introduce a denoising dequantization transform derived from a principled ridge regression objective, creating an explicit, corrective gradient path that makes learning robust to the noise STE bypasses. We extend this to sparsification by treating it as a special form of quantization that zeros out small values. Our unified framework trains models at arbitrary precisions and sparsity levels with off-the-shelf recipes, enabling stable A1W1 and sub-1-bit networks where others falter. It yields state-of-the-art results, mapping efficiency frontiers for modern LLMs and providing a theoretically grounded path to hyper-efficient neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。