LoRA模块相加可高效组合，无需重新训练。
Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models
- 直接相加独立训练的LoRA模块，利用高维正交性实现模块化适配。
- 数学+医学组合降低困惑度9.10%，优于合并数据微调。
- 适合快速部署多领域模型，尤其关注零成本组合与干扰分析。
大规模语言模型的发展依赖于规模，而参数高效微调（PEFT）仅更新少量参数。低秩适应（LoRA）将参数增量表示为两个小矩阵的乘积，使其天然具备可组合性。受叠加原理启发，我们假设在不相交领域上独立训练的LoRA模块近似正交，可通过简单相加组合。使用GPT-2 Small（117M）与LoRA秩4、alpha=64，在三个问答领域（数学、医学、金融）上训练适配器。成对测试中，数学+医学组合使困惑度相对降低9.10%，优于合并数据微调；数学+金融和金融+医学分别提升4.54%和27.56%。所有组合中，LoRA增量之间的余弦相似度均方根与困惑度变化呈正相关且近似线性。朴素求和无需额外训练，可在秒级完成，性能媲美合并数据训练模型，同时揭示高阶组合中的干扰现象。
原文摘要 · Abstract (English)
Recent advances in large language models are driven by scale, while parameter-efficient fine-tuning (PEFT) enables updating only a small fraction of parameters. Low-Rank Adaptation (LoRA) stores parameter deltas as the product of two small matrices, which makes them natural building blocks that can be composed. Motivated by the superposition principle, we hypothesize that independently trained LoRA modules on disjoint domains are approximately orthogonal and can be combined by simple addition. Using GPT-2 Small (117M) with LoRA rank 4 and alpha=64, we train adapters for three QA domains (math, medicine, finance). In pairwise tests, adding Math+Medicine adapters improves perplexity by -9.10% relative to merged-data fine-tuning, while Math+Finance and Finance+Medicine change by +4.54% and +27.56%, respectively. Across combinations, the RMS cosine similarity between LoRA deltas correlates positively and approximately linearly with the change in perplexity. Naive summation requires no additional training, can be applied in seconds, and achieves performance comparable to models trained on merged data, while clarifying when interference appears in higher-order compositions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。