用预测和上下文建模压缩训练检查点，大幅减小存储占用。
An Efficient Compression of Deep Neural Network Checkpoints Based on Prediction and Context Modeling
- 用前一检查点数据做上下文建模，提升算术编码效率
- 结合剪枝与量化，压缩率显著提升，恢复后模型性能几乎无损
- 适合资源受限环境下的模型训练存档与恢复
本文致力于高效压缩神经网络训练过程中不同阶段产生的权重和优化器状态（称为检查点）。首先，提出基于预测的压缩方法，利用先前保存的检查点值进行算术编码中的上下文建模。其次，为提升压缩性能，进一步对检查点数值实施剪枝与量化。实验结果表明，该方法在实现显著比特尺寸缩减的同时，可实现近乎无损的训练恢复，模型性能得以保留，适用于存储受限的场景。
原文摘要 · Abstract (English)
This paper is dedicated to an efficient compression of weights and optimizer states (called checkpoints) obtained at different stages during a neural network training process. First, we propose a prediction-based compression approach, where values from the previously saved checkpoint are used for context modeling in arithmetic coding. Second, in order to enhance the compression performance, we also propose to apply pruning and quantization of the checkpoint values. Experimental results show that our approach achieves substantial bit size reduction, while enabling near-lossless training recovery from restored checkpoints, preserving the model's performance and making it suitable for storage-limited environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。