用预训练模型实现高性能1比特大模型，省去从零训练成本
Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
- 设计渐进式训练机制，平滑转换全精度到二值化权重
- 在多个尺寸LLM上验证，性能超越现有方法且无需重新训练
- 适合关注模型压缩与部署效率的研究者和工程师
1比特大语言模型量化能显著降低存储与计算开销。然而，现有方法通常从零开始训练1比特模型，未能充分利用预训练模型，导致训练成本高且准确率下降明显。我们发现全精度与1比特表示间存在巨大差距，直接适配困难。本文提出一种前向与反向一致的渐进式训练方法，将全精度权重平稳转换为二值化形式。同时引入二值感知初始化和双尺度补偿机制，降低训练难度并提升性能。在不同规模的LLM上实验表明，该方法显著优于现有方案，证明基于预训练模型可实现高性能1比特大模型，无需昂贵的从头训练。
原文摘要 · Abstract (English)
1-bit LLM quantization offers significant advantages in reducing storage and computational costs. However, existing methods typically train 1-bit LLMs from scratch, failing to fully leverage pre-trained models. This results in high training costs and notable accuracy degradation. We identify that the large gap between full precision and 1-bit representations makes naive adaptation difficult. In this paper, we introduce a consistent progressive training for both forward and backward, smoothly converting the full-precision weights into the binarized ones. Additionally, we incorporate binary-aware initialization and dual-scaling compensation to reduce the difficulty of progressive training and improve the performance. Experimental results on LLMs of various sizes demonstrate that our method outperforms existing approaches. Our results show that high-performance 1-bit LLMs can be achieved using pre-trained models, eliminating the need for expensive training from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。