提出PRAC方法，用主成分+随机子空间压缩大模型激活，省内存还快收敛
PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training
- 将激活分解为奇异值分解主子空间和正交随机子空间
- 实现36%内存减少，性能几乎无损，计算开销极低
- 适合大规模语言模型训练中追求内存效率的研究者
大批次训练下，激活值已成为主要的内存瓶颈。现有压缩方法未能利用激活的谱结构，导致收敛慢或压缩率有限。本文揭示算法快速收敛与子空间投影要求之间的关系，指出有效压缩需对原激活提供无偏估计且方差低。提出用于大模型激活压缩的主-随机子空间方法（PRAC），将激活分解为：通过SVD捕获主导信息的主子空间，以及从正交补空间采样的随机子空间以逼近尾部。引入精确缩放因子后，证明在特定条件下PRAC可生成方差最小的无偏梯度估计。预训练与微调任务上的大量实验表明，PRAC实现最高达36%的总内存减少，性能损失可忽略，计算成本极低。
原文摘要 · Abstract (English)
Activations have become the primary memory bottleneck in large-batch LLM training. However, existing compression methods fail to exploit the spectral structure of activations, resulting in slow convergence or limited compression. To address this, we bridge the relationship between the algorithm's fast convergence and the requirements for subspace projection, and show that an effective compression should yield an unbiased estimate of the original activation with low variance. We propose Principal-Random Subspace for LLM Activation Compression (PRAC), which novelly decomposes activations into two components: a principal subspace captured via SVD to retain dominant information, and a random subspace sampled from the orthogonal complement to approximate the tail. By introducing a precise scaling factor, we prove that PRAC yields an unbiased gradient estimator with minimum variance under certain conditions. Extensive experiments on pre-training and fine-tuning tasks demonstrate that PRAC achieves up to 36% total memory reduction with negligible performance degradation and minimal computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。