让MoE模型用低秩适配时，能动态选专家并共享知识。
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
- 用路由激活信息动态控制适配器选择，保留专家专长。
- 全模型共享一个适配器池，实现跨层知识迁移与均衡使用。
- 在多个规模的MoE模型上表现最优，适合高效微调大型专家模型。
混合专家（MoE）架构在大语言模型中广泛应用，但针对MoE模型的参数高效微调（PEFT）研究仍不充分。现有方法要么忽略路由器先验、统一使用适配器，降低效率且易遗忘；要么依赖静态专家选择，限制了每令牌容量和跨专家特征学习。本文首次提出将MoE风格的低秩适配应用于MoE模型：所提方法MoE$^2$-LoRA通过双通道路由条件投影（RCP）模块，利用基础路由器激活信息引导LoRA路由，深度融合预训练专家专长与任务特定适应性。进一步引入全局共享的LoRA专家池，覆盖所有层，实现模型级适应，并涌现层间亲和性与均衡专家利用率。该方法同时具备先验复用、动态适配和全局知识共享优势。在多个不同规模与专家粒度的MoE骨干模型上评估，MoE$^2$-LoRA持续取得最佳下游精度，且保持更强泛化能力。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with uniform adapters, reducing efficiency and risking forgetting, or rely on static expert selection, limiting per-token capacity and cross-expert feature learning. In this paper, we make the first attempt to fine-tune MoE models with MoE-style low-rank adaptation: our method, entitled MoE$^2$-LoRA, deeply couples the pretrained expert specialization with task-specific adaptivity via a dual-channel Routing-Conditioned Projection (RCP) module, which reuses base router activations to inform LoRA routing. We further introduce a single global LoRA expert pool shared across all layers, enabling model-wide adaptation with emergent layer-wise affinities and balanced expert utilization. MoE$^2$-LoRA simultaneously benefits from the advantages of prior reuse, dynamic adapter routing, and model-wide knowledge sharing. Evaluated on multiple MoE backbones with varying scales and expert granularities, MoE$^2$-LoRA consistently achieves state-of-the-art downstream accuracy while retaining stronger general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。