为专家模型设计高效微调方法，仅改少量参数即可提升性能。
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
- 将参数高效微调模块嵌入专家路由机制，灵活适配不同任务。
- 在OLMoE和Mixtral模型上验证，微调后推理准确率显著提升。
- 适合需要低资源微调大模型的研究者与工程师使用。
混合专家（MoE）架构已成为提升Transformer模型扩展性与资源利用率的有力方法。然而，高效微调MoE模型仍鲜有研究。受参数高效微调（PEFT）启发，我们提出一种统一框架，将PEFT模块直接融入MoE机制中。该框架包含多种功能与组合策略的设计维度。基于此，我们构建了适用于MoE模型的参数高效路由微调（PERFT）方法，具备灵活性与可扩展性。在OLMoE-1B-7B与Mixtral-8×7B模型上，针对常识与算术推理任务的实验表明，PERFT具有高效性、可扩展性及独特动态特性。同时，我们对每种设计选择提供了实证分析，以促进MoE与PEFT的更好应用。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) paradigm has emerged as a powerful approach for scaling transformers with improved resource utilization. However, efficiently fine-tuning MoE models remains largely underexplored. Inspired by recent works on Parameter-Efficient Fine-Tuning (PEFT), we present a unified framework for integrating PEFT modules directly into the MoE mechanism. Aligning with the core principles and architecture of MoE, our framework encompasses a set of design dimensions including various functional and composition strategies. By combining design choices within our framework, we introduce Parameter-Efficient Routed Fine-Tuning (PERFT) as a flexible and scalable family of PEFT strategies tailored for MoE models. Extensive experiments on adapting OLMoE-1B-7B and Mixtral-8$\times$7B for commonsense and arithmetic reasoning tasks demonstrate the effectiveness, scalability, and intriguing dynamics of PERFT. Additionally, we provide empirical findings for each specific design choice to facilitate better application of MoE and PEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。