1.58比特模型天然适配稀疏结构，提升大模型效率
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
- 联合1.58比特量化与动态N:M稀疏化，实现稳定训练
- 相同稀疏度下性能损失更小，可承受更高稀疏度
- 适合追求高效推理与训练的大模型部署场景
半结构化N:M稀疏与低比特量化（如1.58比特BitNet）是提升大语言模型效率的两种有效方法，但长期被孤立研究。本文揭示1.58比特BitNet相比全精度模型天然更兼容N:M稀疏。为此提出Sparse-BitNet框架，首次实现1.58比特量化与动态N:M稀疏化的联合应用并保障训练稳定。在多种模型规模与训练策略（稀疏预训练、稠密到稀疏调度）下，1.58比特模型在相同稀疏度下表现优于全精度基线，且在精度崩溃前可容忍更高结构化稀疏度。结合自研稀疏张量核心，训练与推理速度最高提升1.30倍。结果表明，极端低比特量化与半结构化稀疏结合是高效大模型的重要方向。代码已开源。
原文摘要 · Abstract (English)
Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models. To study this effect, we propose Sparse-BitNet, a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30X. These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。