用模块化专家提升语言模型多领域适配效率与知识保留
Scalable Multi-Domain Adaptation of Language Models using Modular Experts
- 引入模块化专家架构，独立训练领域专用模块
- 相比全参微调保留98.35%通用知识，性能接近全参微调
- 支持灵活拆分部署，训练速度提升最高达38%
领域特定适配对预训练语言模型（PLMs）在单一或多个目标任务上的表现至关重要，尤其在资源受限场景如边缘设备中。然而现有方法难以平衡领域性能、通用知识保留与训练推理效率。为此，我们提出模块化领域专家（MoDE），一种基于专家混合的架构，通过模块化、领域专精的专家增强通用PLM。这些专家独立训练，并通过轻量级过程组合。相较于标准低秩适配，每个MoDE专家包含多个Transformer层，可更好适应更多训练样本和更大参数量。评估显示，MoDE在目标性能上媲美全参微调，同时实现1.65%更高的知识保留率；其架构支持灵活分片配置，训练速度相较当前最优分布式配置最高提升38%。
原文摘要 · Abstract (English)
Domain-specific adaptation is critical to maximizing the performance of pre-trained language models (PLMs) on one or multiple targeted tasks, especially under resource-constrained use cases, such as edge devices. However, existing methods often struggle to balance domain-specific performance, retention of general knowledge, and efficiency for training and inference. To address these challenges, we propose Modular Domain Experts (MoDE). MoDE is a mixture-of-experts architecture that augments a general PLMs with modular, domain-specialized experts. These experts are trained independently and composed together via a lightweight training process. In contrast to standard low-rank adaptation methods, each MoDE expert consists of several transformer layers which scale better with more training examples and larger parameter counts. Our evaluation demonstrates that MoDE achieves comparable target performances to full parameter fine-tuning while achieving 1.65% better retention performance. Moreover, MoDE's architecture enables flexible sharding configurations and improves training speeds by up to 38% over state-of-the-art distributed training configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。