用精确优化方法解决低比特神经网络训练难题,实现可证明的高精度
Towards Provable and Scalable Training of Quantized Neural Networks with Ising Optimization
- 将量化网络训练转化为无松弛间隙的二次约束二元优化问题
- 在1.1比特下于二分类Fashion-MNIST达94.95%准确率,抗低比特崩溃
- 适用于需要高可靠性和低资源部署的边缘AI场景
由于非凸损失曲面和离散参数空间,量化神经网络的训练仍具根本挑战。本文提出一种精确的二次约束二元优化(QCBO)框架,具备可证明保证。首先刻画了网络零损失层级集的分层拓扑:一般内部层级为光滑,但全局最优组件即使在过参数化下也可能保持分离。为克服此非凸障碍,将有限深度架构与参数码本、前向区间传播(FIP)有界状态编译为有界QCBO,得到完全正定凸形式,保留全局离散最优解且无松弛间隙。为突破单样本扩展瓶颈,提出样本级分解下界优化(DLBO),将每次伊辛调用从数据集规模降至单样本。DLBO的矩层次形成哈密顿局部性层次,二级提供免辅助的成对QUBO预言机,更高阶则以交互局部性换取更紧边界。通过谱-ADMM与随机舍入严格恢复可行离散参数。在相干伊辛机上的实验显示,1.1比特精度下二分类Fashion-MNIST(外套对凉鞋)准确率达94.95%,表现出对低比特表示崩溃的鲁棒性。多类DLBO在三分类Fashion-MNIST、Wine与Digits数据集上进一步验证了可扩展收敛性。
原文摘要 · Abstract (English)
Training quantized neural networks remains fundamentally challenging due to non-convex loss landscapes and discrete parameter spaces. We introduce an exact Quadratic Constrained Binary Optimization (QCBO) framework with provable guarantees. We first characterize the stratified topology of network zero-loss level sets: generic interior strata are smooth, yet globally optimal components can remain disconnected even under overparameterization. To address this non-convex obstruction, we compile finite-depth architectures with parameter codebooks and Forward Interval Propagation (FIP)-bounded states into bounded QCBOs, yielding an exact completely positive convex formulation that preserves the global discrete optimum with zero relaxation gap. To overcome monolithic sample scaling, we formulate sample-wise Decomposed Lower-Bound Optimization (DLBO) to reduce each Ising call from dataset to single-sample scale. The DLBO moment hierarchy also forms a Hamiltonian-locality hierarchy, with order two giving an auxiliary-free pairwise QUBO oracle and higher orders trading interaction locality for tighter bounds. Strictly feasible discrete parameters are recovered via Spectral--ADMM and randomized rounding. Experiments on a coherent Ising machine achieve $94.95\%$ accuracy on binary Fashion-MNIST (coats vs. sandals) at 1.1-bit precision, demonstrating resilience against low-bit representational collapse. Multi-class DLBO evaluations on 3-class Fashion-MNIST, 3-class Wine, and 3-class Digits further validate scalable convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。