1-bit模型在CPU上实现高速无损推理，大幅降低部署门槛。
1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
- 定制化内核优化1.58位量化模型在CPU上的推理速度。
- x86和ARM架构下速度提升2.37倍至6.17倍。
- 适合资源受限设备本地部署大语言模型的开发者。
近期1比特大语言模型（如BitNet、BitNet b1.58）的发展，为提升大语言模型的运行速度与能效提供了新路径，并支持在多种设备上进行本地部署。本文提出bitnet.cpp，一个专为1比特大语言模型设计的软件栈，开发了一系列内核以支持在CPU上对三值化的BitNet b1.58模型实现快速且无损失的推理。大量实验表明，bitnet.cpp在不同模型规模下，于x86 CPU上获得2.37x至6.17x的加速，在ARM CPU上实现1.37x至5.07x的加速。代码已开源：https://github.com/microsoft/BitNet。
原文摘要 · Abstract (English)
Recent advances in 1-bit Large Language Models (LLMs), such as BitNet and BitNet b1.58, present a promising approach to enhancing the efficiency of LLMs in terms of speed and energy consumption. These developments also enable local LLM deployment across a broad range of devices. In this work, we introduce bitnet.cpp, a tailored software stack designed to unlock the full potential of 1-bit LLMs. Specifically, we develop a set of kernels to support fast and lossless inference of ternary BitNet b1.58 LLMs on CPUs. Extensive experiments demonstrate that bitnet.cpp achieves significant speedups, ranging from 2.37x to 6.17x on x86 CPUs and from 1.37x to 5.07x on ARM CPUs, across various model sizes. The code is available at https://github.com/microsoft/BitNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。