LoFT让低秩微调表现如全量微调，无需调参且不增推理开销。
LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuning
- 通过投影优化器动量与方差到低秩子空间，对齐全量微调动态
- 在多个任务上性能接近全量微调，且收敛更快
- 无需调整缩放因子等超参，适合高效部署场景
大模型常采用参数高效微调方法（如低秩适配器 LoRA）进行下游任务适配，该方法通过引入可训练的低秩矩阵替代全量权重更新，显著减少可训练参数。然而，尽管计算开销小，其精度仍低于全量微调，且收敛较慢。本文提出新方法 LoFT，通过将优化器的内部动态（如 Adam 的一阶和二阶矩）正确投影至相同低秩子空间，实现与全量微调一致的行为。它不仅在低秩子空间中学习权重更新，还精确对齐了优化器状态的演化过程。此设计消除了对额外超参数（如 LoRA 缩放因子 α）的依赖。实验表明，该方法大幅缩小了适配器微调与全量微调之间的性能差距，并持续优于标准 LoRA 方法，且不增加推理成本。
原文摘要 · Abstract (English)
Large pre-trained models are commonly adapted to downstream tasks using parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA), which injects small trainable low-rank matrices instead of updating all weights. While LoRA dramatically reduces trainable parameters with little overhead, it can still underperform full fine-tuning in accuracy and often converges more slowly. We introduce LoFT, a novel low-rank adaptation method that behaves like full fine-tuning by aligning the optimizer's internal dynamics with those of updating all model weights. LoFT not only learns weight updates in a low-rank subspace (like LoRA) but also properly projects the optimizer's first and second moments (Adam's momentum and variance) into the same subspace, mirroring full-model updates. By aligning the low-rank update itself with the full update, LoFT eliminates the need for tuning extra hyperparameters, e.g., the LoRA scaling factor $α$. Empirically, this approach substantially narrows the performance gap between adapter-based tuning and full fine-tuning and consistently outperforms standard LoRA-style methods, all without increasing inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。