分阶段量化训练让模型更高效，16位起步再转1.58位，性能接近全精度。
Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
- 先用16位训练，再过渡到1.58位量化训练，提升效率
- 在11个下游任务中表现优于全程1.58位训练
- 保留优化器状态和渐进量化可减少训练波动
大语言模型（LLMs）训练与推理需要大量资源。量化通过降低模型参数精度，是提升效率与可持续性的有效手段。尽管后训练量化通常实现每参数4-8比特，但近期研究显示，从头训练时使用每权重1.58比特仍能保持模型准确率，显著降低推理时的内存需求与能耗。本文研究一种量化感知预训练策略：模型先以16位精度训练，再转入1.58比特量化感知训练。在11个下游任务上的结果表明，该16至1.58比特训练策略优于全程1.58比特训练，且模型性能更接近16位训练的基准。我们进一步考察了过渡点保留优化器状态及渐进式引入量化强度的影响——两者均能缓解损失突增，但这些优势可通过后续训练补偿。
原文摘要 · Abstract (English)
Large language models (LLMs) require immense resources for training and inference. Quantization, a technique that reduces the precision of model parameters, offers a promising solution for improving LLM efficiency and sustainability. While post-training quantization methods typically achieve 4-8 bits per parameter, recent research suggests that training LLMs with 1.58 bits per weight parameter from scratch can maintain model accuracy while greatly reducing memory requirements and energy consumption at inference time. Here, we investigate a training strategy for quantization-aware pre-training, where the models are first trained with 16-bit precision and then transition into 1.58-bit quantization-aware training. Our results on 11 downstream tasks show that this 16-to-1.58-bit training strategy is preferable over full 1.58-bit training and leaves models closer to those which have undergone 16-bit training. We further investigate the effects of retaining the optimizer state at the transition point and gradually phasing in quantization strength -- finding that both techniques alleviate the magnitude of loss spikes, but also that these effects can be compensated through further training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。