通过列空间投影实现高效微调，理论扎实且效果领先。
PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
- 基于预训练权重的主列空间投影，提供有效归纳偏置。
- 在多任务中优于现有方法，参数量更少仍表现更优。
- 适合资源受限场景下的大模型轻量化微调。
微调大型基础模型对构建特定任务和领域的专家模型至关重要，但全量更新数十亿参数在计算上不可行。采用参数高效微调（PEFT）方法，如低秩适配（LoRA），可显著降低训练成本，并减轻部署时的存储、缓存与服务开销。先前工作（如SVFT）表明，基于奇异值分解（SVD）利用预训练权重几何结构能提升参数效率，但缺乏坚实的理论基础。本文提出一种新方法——列空间投影参数高效微调（PiCa），从理论上证明将梯度投影到预训练权重的主列空间能提供有效的归纳偏置，并通过新颖的权重共享策略进一步提升参数效率。在多种自然语言处理与视觉任务中，PiCa在相近或更小的参数预算下持续优于当前最优基线，展现出理论严谨性与实际有效性。
原文摘要 · Abstract (English)
Fine-tuning large foundation models is essential for building expert models tailored to specialized tasks and domains, but fully updating billions of parameters is computationally prohibitive. Reducing the number of trainable parameters using Parameter-Efficient Fine-Tuning (PEFT), such as Low-Rank Adaptation (LoRA), is therefore crucial not only to reduce training costs but also to mitigate storage, caching, and serving overheads during deployment. Prior works, such as Singular Vectors-guided Fine-Tuning (SVFT), have shown that exploiting the geometry of pre-trained weights based on Singular Value Decomposition (SVD) can significantly improve parameter-efficiency, but they lack a solid theoretical foundation. In this paper, we introduce Parameter-Efficient Fine-Tuning with Column Space Projection (PiCa), a novel theoretically grounded PEFT method. We prove that projecting gradients onto the principal column space of pre-trained weights provides an effective inductive bias for adaptation and further enhance parameter efficiency through a novel weight-sharing strategy. Across diverse NLP and vision tasks, PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets, demonstrating both theoretical rigor and practical effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。