1.58比特量化让多类模型训练高效且性能不降。
When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
- 从感知机到图网络，全链路验证1.58比特训练可行性
- 各类模型在1.58比特下性能持平甚至超越32/16比特基准
- 适合追求极致推理效率的部署场景
当前机器学习模型（如语言模型）虽强大，但训练与推理资源消耗巨大。已有研究证明，仅用三值权重（1.58比特/权值）即可训练出性能可比的解码器单向语言模型，实现高效推理。本文从非Transformer架构出发，探索多层感知机与图神经网络的1.58比特训练；随后拓展至编码器单向及编码器-解码器型变压器语言模型。结果表明，在所有这些设置中，1.58比特训练的性能均与或优于标准的32/16比特模型。
原文摘要 · Abstract (English)
Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。