arXiv:2608.03275cs.CL2026-08

用专家码生成适配参数,实现动态个性化微调。

MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation

论文配图:MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
图 1 · 摘自论文原文
  • 用小型专家码向量代替完整LoRA模块,按需生成参数
  • 在8个常识推理任务上优于静态与现有MoE基线
  • 适合需要灵活适配的多领域应用,如医疗法律

参数高效微调(PEFT)可高效适配大语言模型,但现有基于混合专家(MoE)的PEFT方法通常通过存储多个完整LoRA专家来提升容量,导致适配器存储随专家数量线性增长,并限制适应于固定专家池。本文提出MoEGen,将MoE-based PEFT从专家选择转变为专家条件参数生成。不再为每个专家显式存储完整LoRA模块,而是将每个专家表示为一个小型可学习向量(专家码),根据输入路由这些向量,利用其加权组合作为轻量级超网络的条件,生成输入相关的低秩更新。该设计解耦了专家容量与适配器存储,同时支持实例化条件适配。在八个常识推理基准上的实验表明,MoEGen在三种骨干模型上均持续优于强基线,包括静态和现有MoE-based PEFT方法。此外,在联合医学与法律领域适配中也表现优异。

原文摘要 · Abstract (English)

Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of large language models, but existing MoE-based PEFT methods typically improve capacity by storing multiple full LoRA experts, causing adapter storage to grow linearly with the number of experts and restricting adaptation to a fixed expert pool. We ask whether MoE-based PEFT can produce instance-specific adaptations without explicitly storing a separate LoRA module for each expert. To address this gap, we propose MoEGen, an adaptation framework that shifts MoE-based PEFT from expert selection to expert-conditioned parameter generation. Instead of storing each expert as a full LoRA adapter, MoEGen represents each expert as a small learnable vector, termed an expert code. It routes each input over these vectors and uses their weighted combination to condition a lightweight hypernetwork that generates input-specific low-rank updates. This design decouples expert capacity from adapter storage while enabling instance-conditioned adaptation. Experiments on eight commonsense reasoning benchmarks show consistent improvements over strong static and MoE-based PEFT baselines across three backbones. MoEGen also performs strongly in joint medical and legal-domain adaptation.

参数高效微调混合专家低秩适配动态生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。