提出自适应共享专家机制，提升多任务学习的效率与效果
Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
- 用LoRA构建可共享的专家模块，通过门控权重联合归一化实现动态分配
- 在PASCAL-Context上验证，多种配置下性能均优于现有方法
- 细粒度专家设计在参数量相近时提升知识共享能力，适合资源受限场景
混合专家（MoE）已成为多任务学习（MTL）的重要框架。然而，现有方法多依赖单任务预训练主干，在从单任务到多任务学习（STL到MTL）过渡时存在冗余适配与知识共享效率低的问题。为此，我们提出基于低秩适配（LoRA）的自适应共享专家（ASE），将共享专家的门控权重与稀疏专家联合归一化，支持更平滑的过渡，增强专家专精性与协作。同时，通过增加LoRA专家数量并按比例降低其秩，在不增加参数总量的前提下实现更细粒度的知识共享。在统一训练设置下，于PASCAL-Context基准上大量实验表明，ASE在多种配置中持续提升性能，验证了细粒度设计的有效性。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) has emerged as a powerful framework for multi-task learning (MTL). However, existing MoE-MTL methods often rely on single-task pretrained backbones and suffer from redundant adaptation and inefficient knowledge sharing during the transition from single-task to multi-task learning (STL to MTL). To address these limitations, we propose adaptive shared experts (ASE) within a low-rank adaptation (LoRA) based MoE, where shared experts are assigned router-computed gating weights jointly normalized with sparse experts. This design facilitates STL to MTL transition, enhances expert specialization, and cooperation. Furthermore, we incorporate fine-grained experts by increasing the number of LoRA experts while proportionally reducing their rank, enabling more effective knowledge sharing under a comparable parameter budget. Extensive experiments on the PASCAL-Context benchmark, under unified training settings, demonstrate that ASE consistently improves performance across diverse configurations and validates the effectiveness of fine-grained designs for MTL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。