让专家模型更懂任务语义，动态调整更新强度
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
- 用语义感知路由器精准匹配任务与专家
- 按任务复杂度动态调节专家贡献权重
- 提升多任务学习效果，适合参数高效微调场景
将专家混合（MoE）与低秩适配（LoRA）结合，在提升大语言模型多任务能力方面展现出巨大潜力。然而现有方法存在两大挑战：(1) 当前 MoE-LoRA 方法的路由机制不精确，未能显式匹配输入语义与专家能力，导致专家专业化程度弱；(2) 统一的权重融合策略无法提供自适应的更新强度，忽视不同任务的复杂性差异。为此，我们提出 SAMoRA（语义感知的 LoRA 专家混合），一种专为任务自适应学习设计的参数高效微调框架。具体而言，提出语义感知路由器，显式对齐文本语义与最合适的专家，实现精准路由；设计任务自适应缩放机制，根据特定任务需求动态调节专家贡献；并引入新颖的正则化目标，联合促进专家专业化与有效缩放。在多个多任务基准上的大量实验表明，SAMoRA 显著优于当前最优方法，具备优异的任务泛化能力。代码已开源：https://github.com/boyan-code/SAMoRA
原文摘要 · Abstract (English)
The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。