通过分拆线性层提升低比特语言模型精度与扩展性
pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training
- 将线性层拆分为1比特高效分支和高精度敏感参数分支
- 在LLM中实现亚2比特量化下最佳性能，支持高效扩展
- 适合追求边缘部署低比特模型的开发者与研究者
从头开始的量化感知训练已成为构建极低比特权重（亚2比特）高效大语言模型的有前景方法，可在边缘部署中带来显著优势。然而，现有方法仍难以达到理想准确率与可扩展性。本文识别出参数均质化是关键瓶颈：所有参数敏感度趋于一致，严重限制表达能力。为此，我们提出pQuant，通过将线性层分解为两个专用分支：一个主导的1比特分支用于高效计算，一个紧凑的高精度分支专门保留最敏感参数。通过定制特征缩放，显式引导模型将敏感参数分配至高精度分支。进一步地，将该分支扩展为多个稀疏激活的专家，实现高效容量扩展。大量实验表明，pQuant在极低比特量化下达到当前最优性能。
原文摘要 · Abstract (English)
Quantization-Aware Training from scratch has emerged as a promising approach for building efficient large language models (LLMs) with extremely low-bit weights (sub 2-bit), which can offer substantial advantages for edge deployment. However, existing methods still fail to achieve satisfactory accuracy and scalability. In this work, we identify a parameter democratization effect as a key bottleneck: the sensitivity of all parameters becomes homogenized, severely limiting expressivity. To address this, we propose pQuant, a method that decouples parameters by splitting linear layers into two specialized branches: a dominant 1-bit branch for efficient computation and a compact high-precision branch dedicated to preserving the most sensitive parameters. Through tailored feature scaling, we explicitly guide the model to allocate sensitive parameters to the high-precision branch. Furthermore, we extend this branch into multiple, sparsely-activated experts, enabling efficient capacity scaling. Extensive experiments indicate our pQuant achieves state-of-the-art performance in extremely low-bit quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。