LoRA在特定条件下比全量微调更优，理论揭示其性能优势机制。
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
- 在线性回归框架下对比LoRA与全量微调的泛化误差
- 当任务差异为低秩时，LoRA可实现更低误差，尤其在数据不足时
- 小秩设置反而提升测试精度，适用于资源受限场景
微调通过少量标注数据将预训练模型适配到下游任务。低秩适应（LoRA）是一种高效微调方法，可降低内存与计算开销，且常能达到接近全量微调的性能。尽管广泛应用，但LoRA的理论行为仍不清晰。本文在线性回归设定下研究LoRA，并比较其与全量微调的过拟合风险。分析发现，在过定与欠定两种情形下，存在LoRA优于全量微调的区间。特别地，当预训练与下游任务间的差异本质上为低秩时，LoRA表现更优。进一步表明，LoRA秩的选择影响泛化性能，即使限制模型表达能力，极小秩仍可能提升测试准确率。实验验证了这些理论结果在实际任务中的适用性，表明其洞察力超越线性回归范畴。
原文摘要 · Abstract (English)
Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs while often achieving performance close to full fine-tuning. Despite its widespread use, the theoretical behavior of LoRA is not yet well understood. In this paper, we study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning. Our analysis identifies regimes in which LoRA achieves lower excess risk than full fine-tuning in both overdetermined and underdetermined settings. Specifically, our theory predicts that LoRA can outperform full fine-tuning when the difference between the pretraining and the downstream tasks is effectively low-rank. We further show how the choice of LoRA rank affects generalization performance, explaining why using a very small rank can improve test accuracy in certain settings, even though it limits model expressivity. Finally, we support our theoretical results with experiments on practical tasks, suggesting that the identified tradeoffs and insights extend beyond linear regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。