1-bit大模型用4比特激活，推理更快更省资源。
BitNet a4.8: 4-bit Activations for 1-bit LLMs
- 1-bit权重搭配4比特激活，结合稀疏化与量化降误差。
- 性能媲美BitNet b1.58，推理速度提升，支持INT4/FP4内核。
- 仅激活55%参数，支持3比特键值缓存，适合高效部署。
近期研究如BitNet b1.58展示了1比特大语言模型（LLMs）在降低推理成本的同时保持性能的潜力。本文提出BitNet a4.8，实现1比特LLMs的4比特激活。该方法采用混合量化与稀疏化策略，缓解异常通道带来的量化误差：对注意力与前馈网络层输入使用4比特激活，中间状态则通过稀疏化后进行8比特量化。大量实验表明，BitNet a4.8在训练成本相当的情况下，性能与BitNet b1.58相当，且推理更快，支持INT4/FP4内核。此外，该模型仅激活55%参数，支持3比特键值缓存，进一步提升大规模LLM部署与推理效率。
原文摘要 · Abstract (English)
Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。