边端协同的专家混合模型，高效训练大模型。
Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computing
- 边端设备通过协作分配任务，基于专业能力选择邻居。
- 融合监督与自监督学习，平衡个性化与通用性。
- 适合资源受限场景下大模型的隐私保护训练。
下一代无线网络中，大型人工智能模型（LAMs）正推动移动边缘计算的重大创新。然而，训练LAMs所需的大量计算资源和大规模训练数据，与边缘设备有限的存储和计算能力存在冲突，给模型在边缘的训练与部署带来挑战。本文提出网络化专家混合（NMoE）系统，客户端通过根据各自专长将任务分发给合适邻居，并聚合返回结果实现协同推理。针对NMoE的训练，我们设计了一种联邦学习框架，结合监督与自监督学习，兼顾个性化与泛化能力，同时保障通信效率与数据隐私。通过大量实验验证了所提NMoE系统的有效性，为NMoE训练算法提供了深入见解。
原文摘要 · Abstract (English)
Recent advancements in large artificial intelligence models (LAMs) are driving significant innovations in mobile edge computing within next-generation wireless networks. However, the substantial demands for computational resources and larges-cale training data required to train LAMs conflict with the limited storage and computational capacity of edge devices, posing significant challenges to training and deploying LAMs at the edge. In this work, we introduce the Networked Mixture-of-Experts (NMoE) system, in which clients perform inference collaboratively by distributing tasks to suitable neighbors based on their expertise and aggregate the returned results. For training the NMoE, we propose a federated learning framework that integrates both supervised and self-supervised learning to balance personalization and generalization, while preserving communication efficiency and data privacy. We conduct extensive experiments to demonstrate the efficacy of the proposed NMoE system, providing insights for the NMoE training algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。