让专家模型动态分配专家，提升大模型微调效果
LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
- 用可学习的动态路由替代固定专家选择，支持按令牌和层自适应分配
- 在多个基准上超越现有方法，平均性能领先1.5%以上
- 适合需要高效微调的大模型应用，尤其关注资源优化的场景
近期研究证明，将参数高效微调（PEFT）与专家混合（MoE）结合是适配大语言模型（LLM）下游任务的有效策略。然而，现有方法多依赖传统TopK路由，需精细调参，且为每个令牌固定激活专家数量。本文提出LD-MoLE，一种用于LoRA专家混合的可学习动态路由机制，实现按令牌、分层自适应的专家分配。该方法将非可微的TopK选择替换为可微路由函数与闭式解，使模型能动态决定每层每令牌激活的专家数量。此外，引入解析稀疏性控制目标以调节激活专家数。在Qwen3-1.7B和Llama-3.2-3B模型上的大量实验表明，LD-MoLE在多样化基准上均达到最高平均得分，显著优于当前最优基线，不仅性能更优，还展现出学习令牌依赖与层间专家分配的能力。
原文摘要 · Abstract (English)
Recent studies have shown that combining parameter-efficient fine-tuning (PEFT) with mixture-of-experts (MoE) is an effective strategy for adapting large language models (LLMs) to the downstream tasks. However, most existing approaches rely on conventional TopK routing, which requires careful hyperparameter tuning and assigns a fixed number of experts to each token. In this work, we propose LD-MoLE, a Learnable Dynamic routing mechanism for Mixture of LoRA Experts that enables adaptive, token-dependent, and layer-wise expert allocation. Our method replaces the non-differentiable TopK selection with a differentiable routing function and a closed-form solution. Moreover, our design allows the model to adaptively determine the number of experts to activate for each token at different layers. In addition, we introduce an analytical sparsity control objective to regularize the number of activated experts. Extensive experiments on the Qwen3-1.7B and Llama-3.2-3B models show that LD-MoLE achieves the highest average scores compared to state-of-the-art baselines, across a diverse set of benchmarks. Our method not only achieves superior performance, but also demonstrates the ability to learn token-dependent and layer-wise expert allocation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。