动态压缩大模型训练检查点，16倍压缩不丢精度。
BitSnap: Checkpoint Sparsification and Quantization in LLM Training
- 根据训练阶段自适应稀疏化与量化，兼顾存储与速度。
- 稀疏化达16倍压缩比，量化达2倍压缩比，精度几乎不变。
- 适合大规模模型训练，降低存储与容错成本。
随着大语言模型规模持续增长，高效保存与加载检查点对管理存储、内存占用及训练容错至关重要。现有方法未能全面优化这些方面。本文提出一种新型动态自适应检查点稀疏化与量化方法，针对不同训练阶段和模型架构进行优化。通过系统分析现有有损与无损压缩技术，识别其局限性，并提出在压缩率、速度与精度影响间取得平衡的方案。在多种规模的大语言模型上实验表明，基于位掩码的稀疏化方法实现16倍压缩比且不损失模型精度；基于聚类的量化方法实现2倍压缩比，精度损失极小。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to grow in size and complexity, efficient checkpoint saving\&loading has become crucial for managing storage, memory usage, and fault tolerance in LLM training. The current works do not comprehensively take into account the optimization of these several aspects. This paper proposes a novel checkpoint sparsification and quantization method that adapts dynamically to different training stages and model architectures. We present a comprehensive analysis of existing lossy and lossless compression techniques, identify current limitations, and introduce our adaptive approach that balances compression ratio, speed, and precision impact throughout the training process. Experiments on different sizes of LLMs demonstrate that our bitmask-based sparsification method achieves 16x compression ratio without compromising model accuracy. Additionally, the cluster-based quantization method achieves 2x compression ratio with little precision loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。