通过自适应调整预训练模型的奇异向量和偏置，实现高精度微调。
VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models
- 利用预训练权重的奇异向量和偏置进行自适应微调
- 仅需领先方法1/9的可训练参数，性能接近全量微调
- 适用于低预算场景，覆盖语言与视觉多种任务
主流参数高效微调（PEFT）方法通过在冻结预训练权重 $W$ 上并行添加低秩或稀疏可训练权重来减少可训练参数量。然而这些权重从零开始训练，导致在低预算设置下性能仍落后于全量微调。本文提出VectorFit，一种新参数化方式，通过自适应训练 $W$ 的奇异向量和偏置，高效利用其内在知识。该方法生成的增量权重矩阵 $ΔW$ 具有高秩特性，与全量微调相当。在涵盖19个数据集的广泛实验中，包括自然语言理解与生成、问答、图像分类与生成等任务，VectorFit以9倍更少的可训练参数超越现有基线，在参数效率上表现卓越。
原文摘要 · Abstract (English)
Popular PEFT methods reduce trainable parameter count for fine-tuning by parameterizing new low-rank or sparse trainable weights in parallel to the frozen pre-trained weights $W$. However, these weights are trained from scratch, and there exists a performance gap between these methods and full fine-tuning, especially in low-budget settings. We introduce VectorFit, a new way of parameterization that efficiently utilizes the existing knowledge embedded in $W$ by adaptively training their singular vectors and biases. We show that utilizing the structural and transformational properties of $W$ in this way can lead to high-rank incremental weight matrices $ΔW$, comparable to that of full fine-tuning. VectorFit delivers superior results with 9$\boldsymbol\times$ fewer trainable parameters than the leading PEFT methods. Through comprehensive experiments across 19 datasets covering a wide range of language and vision tasks such as natural language understanding and generation, question answering, image classification, and image generation, we demonstrate that VectorFit surpasses baselines in terms of performance as a function of parameter-efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。