arXiv:2606.14243cs.CL2026-06

提出可独立更新的专家模块,让大模型高效注入外部知识

Decoupled Mixture-of-Experts for Parametric Knowledge Injection

论文配图:Decoupled Mixture-of-Experts for Parametric Knowledge Injection
图 1 · 摘自论文原文
  • 将外部知识转为独立专家模块,路由按需调用
  • 在多个基准上优于检索和适配器基线,提升回答质量
  • 适合需要频繁更新知识的场景,如医疗、法律领域

知识注入旨在为大语言模型注入外部、领域或时效性知识。现有方法在灵活性与整合性之间存在权衡:检索增强生成将知识保留在模型外,仅提供提示级增强;基于微调的方法将新知识编码到共享参数中,但可能引发灾难性遗忘、知识冲突及高昂更新成本。本文提出解耦混合专家(DMoE),一种模块化参数知识注入架构,将专家与路由器从基础模型中解耦。DMoE将外部知识语料转化为可独立更新的专家模块,并使用轻量级不确定性感知路由,在基础模型生成时知识不足时激活相关专家。为支持高效自回归推理,DMoE仅将专家附加到最后一层前馈网络,保留键值缓存复用的同时实现参数级知识增强。在知识密集型基准上的实验表明,DMoE在各项指标上持续优于检索和适配器基线。

原文摘要 · Abstract (English)

Knowledge injection aims to equip large language models (LLMs) with external, domain-specific, or time-sensitive knowledge. Existing approaches typically face a trade-off between flexibility and integration: retrieval-augmented generation keeps knowledge outside the model but only provides prompt-level augmentation, whereas post-training based methods encode new knowledge into shared parameters but may introduce catastrophic forgetting, knowledge conflict, and costly updates. In this paper, we propose Decoupled Mixture-of-Experts (DMoE), a modular architecture for parametric knowledge injection that decouples both experts and the router from the base model. DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts only when the base model lacks sufficient knowledge during generation. To support efficient auto-regressive inference, DMoE attaches experts only to the final-layer feed-forward network, preserving KV-cache reuse while enabling parameter-level knowledge augmentation. Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.

知识注入MoE大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。