arXiv:2510.20673cs.CVcs.LG2025-10NeurIPS被引 4

提出新方法,让模型一次训练支持多精度,提速近8倍。

Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling

  • 用权重偏置校正共享归一化,免去多精度微调
  • 按梯度重要性采样子集,每个精度仅用小样本训练
  • 在多个数据集上提速7.88倍,精度不降

多比特量化网络可通过单一模型支持多种精度级别,实现深度神经网络的灵活部署。然而,现有方法存在显著训练开销:为每个比特宽度均需重复全数据集更新,训练成本随精度数量线性增长。此外,新增或中间精度通常需额外微调阶段,进一步增加负担。为此,我们提出两项技术以大幅降低训练开销而不牺牲模型性能:(i) 权重偏置校正使批归一化共享成为可能,并通过消除不同比特宽度间的量化偏差、对齐激活分布,避免了微调需求;(ii) 比特级核心集采样策略利用隐式知识迁移现象,基于梯度重要性分数选取紧凑且信息丰富的子集,使每个子模型仅在小样本上训练。在CIFAR-10/100、TinyImageNet和ImageNet-1K上,采用ResNet与ViT架构的实验表明,该方法在保持甚至超越基准精度的同时,训练时间最多减少7.88倍。代码已开源:https://github.com/a2jinhee/EMQNet_jk。

原文摘要 · Abstract (English)

Multi-bit quantization networks enable flexible deployment of deep neural networks by supporting multiple precision levels within a single model. However, existing approaches suffer from significant training overhead as full-dataset updates are repeated for each supported bit-width, resulting in a cost that scales linearly with the number of precisions. Additionally, extra fine-tuning stages are often required to support additional or intermediate precision options, further compounding the overall training burden. To address this issue, we propose two techniques that greatly reduce the training overhead without compromising model utility: (i) Weight bias correction enables shared batch normalization and eliminates the need for fine-tuning by neutralizing quantization-induced bias across bit-widths and aligning activation distributions; and (ii) Bit-wise coreset sampling strategy allows each child model to train on a compact, informative subset selected via gradient-based importance scores by exploiting the implicit knowledge transfer phenomenon. Experiments on CIFAR-10/100, TinyImageNet, and ImageNet-1K with both ResNet and ViT architectures demonstrate that our method achieves competitive or superior accuracy while reducing training time up to 7.88x. Our code is released at https://github.com/a2jinhee/EMQNet_jk.

量化训练加速多精度核心集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。