通过冻结空空间低秩矩阵，提升微调效率与效果。
Null-LoRA: Low-Rank Adaptation on Null Space
- 在预训练模型的空空间内进行低秩微调，减少冗余。
- 参数更少的情况下，在图文检索与视觉问答任务上超越现有方法。
- 适合追求高效微调且资源受限的研究者使用。
参数高效微调方法在将大规模模型适配到下游任务中广受欢迎，尤其是LoRA及其变体。现有方法在完整参数空间上进行低秩适应，但子空间内的微调同样有效。受预训练模型存在非平凡空空间的启发，本文提出基于空空间的低秩适应（Null-LoRA）。Null-LoRA通过冻结部分低秩矩阵，有效降低冗余并提升有效秩。为进一步提高参数效率，它将全部增量更新约束在空空间内，最大化增量更新对新任务范式的适应能力。在图像-文本检索和视觉问答任务上的大量实验表明，相比现有最优方法，Null-LoRA以更少参数实现更好性能。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods have gained considerable popularity for adapting large-scale models to downstream tasks, particularly LoRA and its variants. Existing methods perform low-rank adaptation over the full parameter space. However, fine-tuning within a subspace can achieve comparable effectiveness. Inspired by the observation that pre-trained models possess non-trivial null spaces, we propose Null-space based Low-Rank Adaptation (Null-LoRA). Null-LoRA effectively reduces redundancy and enhances effective rank by freezing portions of the low-rank matrices. To further improve parameter efficiency, Null-LoRA constrains the entire incremental update within the null space, maximizing the utilization of incremental updates to adapt to new task paradigms. Null-LoRA surpasses the state of the art with fewer parameters in extensive experiments across image-text retrieval and visual question answering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。