通过奇异值分解与输出维度初始化，实现低秩微调的高效参数优化
OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
- 基于SVD分解预训练权重，仅优化输出维度向量
- 训练参数量线性增长，最高秩下仍保持高效
- 在数学与常识推理任务中超越或媲美现有方法
大型语言模型（LLMs）的微调因规模庞大而日益困难，计算成本高昂。参数高效微调（PEFT）虽为替代方案，但依然需要大量资源。本文提出OSoRA（输出维度与奇异值初始化的低秩适应），一种新型的LLM PEFT方法。OSoRA在低秩适应（LoRA）基础上，将奇异值分解（SVD）与可学习缩放向量统一整合。其先对预训练权重矩阵进行SVD，再在训练中优化输出维度向量，同时冻结对应奇异向量矩阵。该方法显著降低微调时的计算资源需求，大幅减少可训练参数数量。在数学推理、常识推理等多类基准测试中，OSoRA表现与当前最优方法（如LoRA、VeRA）相当甚至更优，且在秩升至更高维度时仍保持线性参数增长。消融实验表明，联合训练奇异值与输出维度向量对性能至关重要。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) has become increasingly challenging due to their massive scale and associated computational costs. Parameter-Efficient Fine-Tuning (PEFT) methodologies have been proposed as computational alternatives; however, their implementations still require significant resources. In this paper, we present OSoRA (Output-Dimension and Singular-Value Initialized Low-Rank Adaptation), a novel PEFT method for LLMs. OSoRA extends Low-Rank Adaptation (LoRA) by integrating Singular Value Decomposition (SVD) with learnable scaling vectors in a unified framework. It first performs an SVD of pre-trained weight matrices, then optimizes an output-dimension vector during training, while keeping the corresponding singular vector matrices frozen. OSoRA substantially reduces computational resource requirements by minimizing the number of trainable parameters during fine-tuning. Comprehensive evaluations across mathematical reasoning, common sense reasoning, and other benchmarks demonstrate that OSoRA achieves comparable or superior performance to state-of-the-art methods like LoRA and VeRA, while maintaining a linear parameter scaling even as the rank increases to higher dimensions. Our ablation studies further confirm that jointly training both the singular values and the output-dimension vector is critical for optimal performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。