用轻量适配器让旧大模型变聪明，不训练全模型也能提升性能。
MoIN: Mixture of Introvert Experts to Upcycle an LLM
- 将训练数据分组，每组训练一个轻量适配器专家。
- 推理时只激活最相关专家，效果优于原模型。
- 专家独立训练推理，可并行部署，适合资源有限场景。
本文旨在在不需持续预训练整个模型的前提下，提升现有大型语言模型的性能。方法是将预训练数据划分为语义相关的子集，并为每个子集训练一个专家。这些专家以轻量级适配器的形式附加在冻结的基础模型之上。推理时，输入查询首先被路由到最相关的专家，该专家随后加载到基础模型上完成前向传播。与典型Mixture of Experts（MoE）模型不同，本方法中的专家不与其他专家协作处理单一查询，因此我们称之为“内向型”专家。通过冻结基础模型并保持专家为轻量适配器，训练和推理均可实现极致并行化：所有专家可无通信地并行训练；推理时也可将专家分布于不同GPU，按请求路由至对应设备，实现高效并行。我们实现了该方法的原型验证了其有效性。
原文摘要 · Abstract (English)
The goal of this paper is to improve (upcycle) an existing large language model without the prohibitive requirements of continued pre-training of the full-model. The idea is to split the pre-training data into semantically relevant groups and train an expert on each subset. An expert takes the form of a lightweight adapter added on the top of a frozen base model. During inference, an incoming query is first routed to the most relevant expert which is then loaded onto the base model for the forward pass. Unlike typical Mixture of Experts (MoE) models, the experts in our method do not work with other experts for a single query. Hence, we dub them "introvert" experts. Freezing the base model and keeping the experts as lightweight adapters allows extreme parallelism during training and inference. Training of all experts can be done in parallel without any communication channels between them. Similarly, the inference can also be heavily parallelized by distributing experts on different GPUs and routing each request to the GPU containing its relevant expert. We implement a proof-of-concept version of this method and show the validity of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。