通过主子空间适配，实现高效且保持语义的模型微调。
Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
- 将正交变换限制在预训练权重的主子空间内，提升表达能力。
- 在35个任务上验证，参数、内存和计算三方面均显著优化。
- 适合需要高效微调大模型的研究者和工程师使用。
随着模型参数规模迅速增长，参数高效微调（PEFT）在计算资源受限下已成为适应下游任务的关键。现有正交微调方法虽能保留预训练模型的语义表示，但在参数量、内存与计算开销之间难以兼顾表达力与效率。为此，我们提出高效正交微调的主子空间适配方法（PSOFT），将正交变换限制在预训练权重的主子空间内。具体地，通过矩阵分解构建该子空间，实现更高有效秩的兼容变换；建立严格保持子空间几何结构的理论条件，确保语义稳定性；引入可高效调整的向量，在训练中逐步放松正交性以增强适应性。在四个代表性模型上的35个自然语言处理与计算机视觉任务上进行广泛实验，结果表明PSOFT能同时实现语义保留、表达力与多维度效率的平衡，具备实用性和可扩展性。代码已公开于https://github.com/fei407/PSOFT。
原文摘要 · Abstract (English)
Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parameter counts, memory, and computation. To overcome this limitation, we propose efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations with higher effective rank, establishes a theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability. Extensive experiments on 35 NLP and CV tasks across four representative models demonstrate that PSOFT offers a practical and scalable solution to simultaneously achieve semantic preservation, expressiveness, and multi-dimensional efficiency in PEFT. The code is publicly available at https://github.com/fei407/PSOFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。