通过压缩梯度动量实现高效大模型微调，内存占用更低且性能不降。
MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation
- 用动量低秩压缩替代传统参数更新，减少内存消耗。
- 在低秩（如r=4）下表现接近全参数微调，跨优化器泛化好。
- 适合资源受限场景下的大模型高效微调，尤其适合部署端。
随着大语言模型规模增大，全参数微调带来巨大内存开销。本文提出一种名为MLorc的新型内存高效训练范式，核心思想是在训练过程中对矩阵参数的动量进行压缩与重构，以降低内存消耗。相比LoRA，MLorc不强制权重更新矩阵的固定秩，支持全参数学习；相比GaLore，MLorc直接压缩动量而非梯度，更完整保留全参数微调的训练动态。我们提供了在温和假设下的收敛性理论保证。实验表明，MLorc在多种任务中持续优于其他内存高效方法，在小秩（如 $r=4$）下性能可媲美甚至超过全参数微调，且对不同优化器具有良好的泛化能力，同时保持时间与内存效率。
原文摘要 · Abstract (English)
With increasing size of large language models (LLMs), full-parameter fine-tuning imposes substantial memory demands. To alleviate this, we propose a novel memory-efficient training paradigm called Momentum Low-rank compression (MLorc). The key idea of MLorc is to compress and reconstruct the momentum of matrix parameters during training to reduce memory consumption. Compared to LoRA, MLorc avoids enforcing a fixed-rank constraint on weight update matrices and thus enables full-parameter learning. Compared to GaLore, MLorc directly compress the momentum rather than gradients, thereby better preserving the training dynamics of full-parameter fine-tuning. We provide a theoretical guarantee for its convergence under mild assumptions. Empirically, MLorc consistently outperforms other memory-efficient training methods, matches or even exceeds the performance of full fine-tuning at small ranks (e.g., $r=4$), and generalizes well across different optimizers, all while not compromising time or memory efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。