NeuZip动态压缩模型权重，训练内存减半且性能不降
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
- 基于浮点数熵的动态权重压缩机制
- 训练Llama-3 8B内存从31GB降至16GB以下
- 适合资源受限设备的高效训练与推理
神经网络性能随参数增加而提升，但训练与推理时受限于设备内存。现有量化等技术虽缓解内存压力，但导致性能下降。本文提出NeuZip，一种基于神经网络中浮点数熵的权重压缩方案。该方法可在不损失性能的前提下实现内存高效的训练与推理。显著地，使用NeuZip将训练Llama-3 8B模型的内存占用从31GB降低至16GB以下，同时保持训练动态完全不变。在推理阶段，内存使用减少超过一半，且性能近乎无损。代码已公开。
原文摘要 · Abstract (English)
The performance of neural networks improves when more parameters are used. However, the model sizes are constrained by the available on-device memory during training and inference. Although applying techniques like quantization can alleviate the constraint, they suffer from performance degradation. In this work, we introduce NeuZip, a new weight compression scheme based on the entropy of floating-point numbers in neural networks. With NeuZip, we are able to achieve memory-efficient training and inference without sacrificing performance. Notably, we significantly reduce the memory footprint of training a Llama-3 8B model from 31GB to less than 16GB, while keeping the training dynamics fully unchanged. In inference, our method can reduce memory usage by more than half while maintaining near-lossless performance. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。