通过压缩激活值，显著降低大模型训练的显存占用。
CompAct: Compressed Activations for Memory-Efficient LLM Training
- 用随机投影压缩反向传播所需的激活值，减少显存占用。
- 预训练时显存降25-30%，微调时降50%。
- 适用于大模型训练，尤其适合资源受限场景。
我们提出CompAct，一种可将大模型预训练时峰值显存降低25-30%、微调时降低50%的压缩激活技术。显存瓶颈是大模型训练的主要限制因素，现有方法多聚焦于模型参数或优化器内存，但未解决训练过程中最大的内存开销——计算图存储（用于反向传播）。CompAct通过存储低秩压缩的激活值替代原始数据，大幅减少所需显存，且采用随机投影矩阵，无额外内存开销。相比现有预训练与微调方法，CompAct在计算-性能权衡上实现显著提升，预计在更大模型上效果更优。
原文摘要 · Abstract (English)
We introduce CompAct, a technique that reduces peak memory utilization on GPU by 25-30% for pretraining and 50% for fine-tuning of LLMs. Peak device memory is a major limiting factor in training LLMs, with various recent works aiming to reduce model memory. However most works don't target the largest component of allocated memory during training: the model's compute graph, which is stored for the backward pass. By storing low-rank, compressed activations to be used in the backward pass we greatly reduce the required memory, unlike previous methods which only reduce optimizer overheads or the number of trained parameters. Our compression uses random projection matrices, thus avoiding additional memory overheads. Comparisons with previous techniques for either pretraining or fine-tuning show that CompAct substantially improves existing compute-performance tradeoffs. We expect CompAct's savings to scale even higher for larger models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。