不依赖梯度训练量化神经网络，显著降低能耗与更新次数。
Gradient-Free Training of Quantized Neural Networks
- 摒弃梯度优化，采用新启发式框架避免完整参数更新
- 在标准数据集上性能接近全精度梯度训练，能耗降低3倍
- 适合对能效敏感的边缘设备部署
训练神经网络需要大量计算资源和能源。尽管混合精度和量化感知训练可减少位宽，但仍严重依赖计算开销大的梯度优化。本文提出范式转变：完全消除梯度。虽然在有限量化空间中无梯度寻优看似更简单，但我们理论上证明该问题在简单情形下仍是NP难，即便连续情况可高效求解。为此，我们提出一种新颖启发式优化框架，避免完整权重更新，显著提升效率。实验证明，本方法在标准数据集和架构上性能接近全精度梯度训练，同时能耗降低至1/3，参数更新次数减少至1/5。
原文摘要 · Abstract (English)
Training neural networks requires significant computational resources and energy. Methods like mixed-precision and quantization-aware training reduce bit usage, yet they still depend heavily on computationally expensive gradient-based optimization. In this work, we propose a paradigm shift: eliminate gradients altogether. One might hope that, in a finite quantized space, finding optimal weights with out gradients would be easier but we theoretically prove that this problem is NP-hard even in simple settings where the continuous case is efficiently solvable. To address this, we introduce a novel heuristic optimization framework that avoids full weight updates and significantly improves efficiency. Empirically, our method achieves performance comparable to that of full-precision gradient-based training on standard datasets and architectures, while using up to 3x less energy and requiring up to 5x fewer parameter updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。