用廉价设备分训专家模型，再合并调优，大幅降本提效。
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
- 拆解专家模型为多个子模型，分块独立训练
- 在多数据子集上训练，性能媲美全参数训练
- 适合预算有限但想训练大模型的研究者
大规模混合专家（MoE）模型训练通常依赖高内存、高带宽GPU（如A100），成本高昂。相比之下，低成本硬件受限于内存和带宽，难以直接用于大模型训练。为此，我们提出MoE-DisCo（解耦聚类与协调的混合专家），一种分阶段训练框架。该方法将MoE模型分解为多个稠密子模型，每个包含共享主干和单个专家，并使用无监督聚类划分训练数据。各子模型在分配的数据子集上独立并行训练，仅使用低成本设备，无需设备间通信。随后，所有专家整合为完整MoE模型，在高内存、高带宽GPU上进行短时全局微调。实验表明，该方法在多个下游任务、损失函数和困惑度（PPL）上达到甚至超越全参数训练效果，同时在Qwen1.5-MoE-2.7B和Llama-MoE-3.5B上降低47.6%至69.5%的训练成本。
原文摘要 · Abstract (English)
Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardware is low-cost but constrained by memory capacity and bandwidth, making it unsuitable for direct LLM training. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination), a staged training framework. MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset using low-cost devices, without any inter-device communication. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period on high-memory, high-bandwidth GPUs. Experiments show that our method matches or even surpasses full-parameter training in performance across multiple downstream tasks, loss function, and perplexity (PPL), while reducing training cost by 47.6 percent to 69.5 percent on Qwen1.5-MoE-2.7B and Llama-MoE-3.5B across different datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。