冻结专家模型的前馈层，实现高效多领域模型训练。
MoFE: Mixture of Frozen Experts Architecture
- 冻结MoE中的前馈网络，仅微调少量参数。
- 训练效率显著提升，多领域适配能力增强。
- 适合资源受限场景下的高效模型部署。
我们提出混合冻结专家（MoFE）架构，融合参数高效微调（PEFT）与专家混合（MoE）机制，以提升训练效率和模型可扩展性。通过在MoE框架中冻结前馈网络（FFN）层，MoFE大幅减少可训练参数数量，提高训练效率，同时仍能有效传递专家模型的知识，促进多领域模型的构建。实验评估了性能与效率的权衡，对比了其他PEFT方法，分析了组件模型中领域专长的影响，并确定了最优训练策略。结果表明，尽管性能可能存在轻微损失，但效率提升显著，使MoFE成为资源受限环境下实际应用的合理选择。
原文摘要 · Abstract (English)
We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。