让1比特大模型用4比特激活，显著降低内存和计算开销
BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
- 用哈达玛变换预处理激活值，缓解异常值问题
- 4比特激活训练下性能几乎无损，比8比特还省资源
- 适合追求极致部署效率的低比特大模型研究者
高效部署1比特大语言模型受限于激活值异常点，导致低位宽量化困难。本文提出BitNet v2，首次实现1比特大模型原生4比特激活量化。针对注意力与前馈网络中的激活异常值，提出H-BitLinear模块，在量化前引入在线哈达玛变换，将尖锐的激活分布转化为更接近高斯分布的形式，适合低位宽表示。实验表明，从头训练的BitNet v2使用8比特激活时，性能达到BitNet b1.58水平；关键的是，使用原生4比特激活训练时性能几乎无损，大幅降低批量推理的内存占用与计算成本。
原文摘要 · Abstract (English)
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。