将大模型压缩至1.58比特，实现高效推理。
BitNet Distillation
- 用三元权重(-1,0,1)替代原模型参数，大幅降低存储需求。
- 在下游任务上性能接近全精度模型，内存减少10倍，推理快2.65倍。
- 适合资源受限场景下的大模型部署，如边缘设备。
本文提出BitNet Distillation（BitDistill），一种轻量级微调流程，可将现成的全精度大语言模型（如Qwen）转化为1.58比特精度（即三元权重{-1, 0, 1}）的模型，以实现特定下游任务的高性能表现，同时计算开销极低。BitDistill结合三项关键技术：BitNet中提出的SubLN模块、基于MiniLM的多头注意力蒸馏，以及持续预训练——该步骤作为关键预热机制，缓解了微调后1.58比特模型在特定任务上与全精度模型之间的性能差距问题。实验表明，BitDistill在模型规模相近的情况下性能媲美全精度模型，同时在CPU上实现最高达10倍的内存节省和2.65倍的推理加速。代码已公开于https://github.com/microsoft/BitNet。
原文摘要 · Abstract (English)
In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。