改进LoRA混合专家模型，提升微调时的稳定性和表达能力。
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
- 用多空间投影策略优化MoE-LoRA训练，增强特征学习
- 在SGD和AdamW下均显著提升稳定性与性能
- 适合需要高效微调大模型的研究者和工程师
为简化基础模型的微调流程，低秩适配器（LoRA）已在指令微调和领域适应等多个领域广泛采用。其核心思想是将全秩矩阵分解为两个低秩矩阵的乘积，从而降低存储开销并加速训练。为进一步突破LoRA的表达能力瓶颈，引入了混合专家（MoE）机制以集成多个LoRA适配器，该方法在多个下游任务中表现明显提升。然而，现有混合低秩适配器（MoE-LoRA）在微调与推理过程中仍存在鲁棒性不足的问题。受黎曼预处理方法启发，本文提出一种新的训练策略，将LoRA视为子空间投影器，通过多空间投影稳定并增强特征学习过程。在SGD和AdamW优化器上的实验验证了该方法的有效性。源代码已公开于https://github.com/THUDM/MoELoRA_Riemannian。
原文摘要 · Abstract (English)
In order to streamline the fine-tuning of foundation models, Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation. The underlying concept of LoRA involves decomposing a full-rank matrix into the product of two lower-rank matrices, which reduces storage consumption and accelerates the training process. Furthermore, to address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters. The integration of LoRA experts leads to a visible improvement across several downstream scenes. However, the mixture of LoRAs (MoE-LoRA) still exhibits its low robustness during tuning and inferring. Inspired by the Riemannian Preconditioners which train LoRA as a sub-space projector, we propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections. Examinations on SGD and AdamW optimizers demonstrate the effectiveness of our methodology. Source code is available at https://github.com/THUDM/MoELoRA_Riemannian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。