用QR分解构造正交基，让LoRA只训练系数，大幅减少参数量。
QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models
- 用带列主元的QR分解提取权重矩阵正交基，更新仅需训练系数
- 仅601个参数即达全量微调性能，比标准LoRA少77倍
- 适合资源受限场景下的大模型高效微调
大型语言模型规模不断增长，亟需参数高效的微调技术。低秩适配（LoRA）通过在预训练权重上施加低秩更新来减少可训练参数。传统LoRA直接学习两个更新矩阵，而近期一些变体先通过奇异值分解（SVD）初始化矩阵——该操作在大模型上代价高昂，且奇异向量难以解释。本文提出使用带列主元的QR分解从预训练权重矩阵中提取正交基，并将LoRA更新表示为这些基向量的线性组合，仅训练标量系数，使适配结构清晰且参数量大幅降低。在GLUE任务上的实验表明，QR-LoRA在仅601个参数下达到或超过全量微调、标准LoRA及基于SVD的LoRA性能，相比全量微调减少超1000倍参数，比典型LoRA设置少77倍。
原文摘要 · Abstract (English)
The growing scale of Large Language Models (LLMs) has necessitated the development of parameter-efficient fine-tuning techniques. Low-Rank Adaptation (LoRA) has emerged as a promising approach, reducing the number of trainable parameters by applying low-rank updates to pretrained weights. While standard LoRA learns both update factors directly, several recent variants first initialize those matrices via an SVD of the pretrained weights -- an operation that can be expensive on large models and yields singular vectors that are not always easy to interpret. In this work, we extract an orthonormal basis from the pretrained weight matrix using QR decomposition with column pivoting, and then express the LoRA update as a linear combination of these basis vectors -- training only the scalar coefficients, which imposes clear structure on adaptation and drastically reduces parameter count. Experiments across GLUE tasks show that QR-LoRA matches or exceeds the performance of full fine-tuning, standard LoRA, and SVD-LoRA (LoRA with update matrices initialized via singular value decomposition) with as few as 601 parameters -- a reduction of over 1000x compared to full fine-tuning and 77x fewer than typical LoRA setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。