SOS-LoRA通过多专家正交分解提升低秩微调性能。
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
- 将低秩更新拆分为多个静态专家，用正交初始化避免方向耦合。
- 在多个基准上优于同预算的LoRA基线，尤其在数学和推理任务中提升明显。
- 适合需要高效微调大模型且避免干扰的场景，代码开源可复现。
低秩适配（LoRA）是一种广泛使用的大型语言模型参数高效微调方法。在固定秩预算下，传统LoRA通过单一低维输入路径参数化每个权重，可能因共享输入方向导致异质行为耦合，并在优化过程中引发干扰。本文提出静态正交子空间低秩适配（SOS-LoRA），作为即插即用的扩展，将秩为rtot的更新重参数化为K个静态（始终开启、非路由）低秩专家之和。SOS-LoRA（i）在专家间分配总秩，（ii）采用固定多尺度缩放方案以促进尺度分离的优化动态，（iii）通过跨专家正交初始化与轻量级正则化鼓励多样化的输入方向。该方法保持完全可合并性，合并后不增加推理时参数或延迟。在推理与知识密集型任务（Llama 2/3）、基于编码器的自然语言理解（GLUE）以及数学推理（GSM8K/MATH）上的实验表明，其性能一致优于同预算的LoRA基线及近期变体。代码已公开于https://github.com/llm172/sos-lora。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。