arXiv:2501.00365cs.LGcs.AI2025-01综述被引 63

LoRA让大模型高效适配新任务,只需少量参数更新。

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

  • 用低秩矩阵分解替代全量微调,大幅减少参数改动。
  • 在多个领域实现与全量微调相当性能,参数量仅为其1%-2%。
  • 适合资源有限的团队快速部署大模型到具体任务。

基础模型——在多样且大规模数据集上训练的大规模神经网络——的快速发展彻底改变了人工智能,在自然语言处理、计算机视觉及科学发现等领域实现了前所未有的突破。然而,这些模型动辄数十亿甚至万亿级的参数量,使其在适配下游特定任务时面临巨大挑战。低秩适应(LoRA)作为一种极具前景的方法,通过参数高效的方式对基础模型进行微调,计算开销极小。本综述首次系统性地回顾了LoRA技术在大语言模型之外的通用基础模型中的应用,涵盖最新进展、新兴前沿及多领域应用。最后,本文讨论了理论理解、可扩展性与鲁棒性等方面的关键挑战与未来研究方向。该综述为从事高效基础模型适配的研究人员与实践者提供了宝贵资源。

原文摘要 · Abstract (English)

The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation.

低秩适应大模型微调参数效率基础模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。