高效微调大模型，用极少参数实现高性能。
Parameter-Efficient Fine-Tuning for Foundation Models
- 仅修改少量参数，即可适配不同大模型任务。
- 支持语言、图像、多模态等多样大模型微调。
- 适合想快速部署大模型的开发者与研究者。
本文综述了基础模型（FMs）领域的参数高效微调（PEFT）技术。PEFT是一种低成本微调方法，通过最小化参数量和计算复杂度，同时保持下游任务的最优性能。基础模型如ChatGPT、DALL-E、LLaVA分别擅长语言理解、生成任务和多模态任务，其训练数据涵盖文本、图像和视频。模型多样性促使多种适配策略的发展。本综述系统梳理了各类基础模型中PEFT的关键类别与核心机制，揭示技术趋势。同时展示其在多类模型中的最新应用，凸显其通用性。最后指出未来优化方向。文末附有完整论文列表（https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models），为初学者与专家提供实用参考。
原文摘要 · Abstract (English)
This survey delves into the realm of Parameter-Efficient Fine-Tuning (PEFT) within the context of Foundation Models (FMs). PEFT, a cost-effective fine-tuning technique, minimizes parameters and computational complexity while striving for optimal downstream task performance. FMs, like ChatGPT, DALL-E, and LLaVA specialize in language understanding, generative tasks, and multimodal tasks, trained on diverse datasets spanning text, images, and videos. The diversity of FMs guides various adaptation strategies for PEFT. Therefore, this survey aims to provide a comprehensive overview of PEFT techniques applied to diverse FMs and address critical gaps in understanding the techniques, trends, and applications. We start by providing a detailed development of FMs and PEFT. Subsequently, we systematically review the key categories and core mechanisms of PEFT across diverse FMs to offer a comprehensive understanding of trends. We also explore the most recent applications across various FMs to demonstrate the versatility of PEFT, shedding light on the integration of systematic PEFT methods with a range of FMs. Furthermore, we identify potential research and development directions for improving PEFTs in the future. This survey provides a valuable resource for both newcomers and experts seeking to understand and use the power of PEFT across FMs. All reviewed papers are listed at \url{https://github.com/THUDM/Awesome-Parameter-Efficient-Fine-Tuning-for-Foundation-Models}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。