提出新方法，训练大模型时省内存还更快更准。
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection
- 通过感知梯度投影间相关性，优化低秩压缩策略。
- 在LLaMA-1B上节省61%内存，仅多2%时间，效果媲美AdamW。
- 适合资源有限但需高效训练大模型的研究者或工程师。
大规模视觉与多模态神经网络训练需要大量内存，主要源于优化器状态的存储。尽管参数高效方法如LoRA可降低内存开销，但受限于低秩更新常导致性能不佳。现有低秩梯度投影方法（如GaLore、Flora）通过奇异值分解或随机投影将梯度和动量投影至低秩空间以减少内存占用，但忽略了不同投影间的相关性，导致性能下降，且投影策略计算开销高。本文提出COAP（Correlation-Aware Gradient Projection），一种兼顾低内存与低计算成本的训练方法，在多种视觉、语言及多模态任务中均表现出色。在LLaMA-1B上，其优化器内存减少61%，额外耗时仅2%，且达到与AdamW相当的PPL；在8位量化下，对LLaVA-v1.5-7B微调，内存减少81%，速度提升4倍，同时精度更高。
原文摘要 · Abstract (English)
Training large-scale neural networks in vision, and multimodal domains demands substantial memory resources, primarily due to the storage of optimizer states. While LoRA, a popular parameter-efficient method, reduces memory usage, it often suffers from suboptimal performance due to the constraints of low-rank updates. Low-rank gradient projection methods (e.g., GaLore, Flora) reduce optimizer memory by projecting gradients and moment estimates into low-rank spaces via singular value decomposition or random projection. However, they fail to account for inter-projection correlation, causing performance degradation, and their projection strategies often incur high computational costs. In this paper, we present COAP (Correlation-Aware Gradient Projection), a memory-efficient method that minimizes computational overhead while maintaining training performance. Evaluated across various vision, language, and multimodal tasks, COAP outperforms existing methods in both training speed and model performance. For LLaMA-1B, it reduces optimizer memory by 61% with only 2% additional time cost, achieving the same PPL as AdamW. With 8-bit quantization, COAP cuts optimizer memory by 81% and achieves 4x speedup over GaLore for LLaVA-v1.5-7B fine-tuning, while delivering higher accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。