arXiv:2510.16250cs.LGstat.ML2025-10

证明除最后一层外,全模型可一键位压缩而不损失泛化性能。

One-Bit Quantization for Random Features Models

  • 在随机特征模型中,除最后一层外所有层权重可一比特量化
  • 渐近条件下,量化后泛化误差与全精度模型几乎无差别
  • 为神经网络压缩提供理论支持,适合边缘设备部署场景

近期神经网络发展带来了巨大的计算和内存需求,推动了一比特权重压缩的研究,以实现资源受限设备上的高效推理。然而,这种压缩的理论基础仍不清晰。本文针对随机特征模型(一种对应于具有随机表示的神经网络的简化框架)分析了一比特量化,证明在渐近条件下,仅对最后一层以外的所有层进行量化,不会导致泛化误差的增加。我们的结果为神经网络压缩提供了理论依据。同时,实验表明,即使在笔记本电脑的GPU上,一比特量化也能显著提升随机特征模型的推理速度,验证了其实际效益。此外,我们还给出了任意层数随机特征模型的泛化误差的渐近精确刻画。据我们所知,该分析比现有相关文献中的结果更具普遍性。

原文摘要 · Abstract (English)

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.

一比特量化随机特征理论分析模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。