PC-MoE让多方在低显存下安全协作训练大模型，性能接近中心化方案。
PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
- 利用MoE稀疏性实现分布式协同训练，本地保留数据与梯度
- 7个基准测试中性能接近甚至超过集中式模型，显存降低近70%
- 适合资源有限且注重隐私的机构联合训练大模型
Mixture-of-Experts（MoE）因其在大语言模型（LLMs）中的成功应用而日益流行。本文提出隐私保护协同MoE（PC-MoE），利用MoE架构的稀疏性，实现内存高效的去中心化协同训练，使多个显存和数据资源有限的参与方能够共同训练出比各自独立训练更强大的大语言模型。同时，该方法通过将训练数据、部分前向信号和梯度保留在各参与方本地，保护了各参与方的训练数据隐私。设计上，PC-MoE巧妙结合了分布式计算的优势与强保密性保障。不同于多数隐私保护方案以牺牲任务准确率为代价换取保密性，本框架打破了这一权衡：在七个主流大模型基准测试中，其性能几乎与完全集中式模型相当（甚至有时超越），峰值显存减少近70%，且对重建攻击具有完全鲁棒性。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) has been gaining popularity due to its successful adaptation to large language models (LLMs). In this work, we introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE), which leverages the sparsity of the MoE architecture for memory-efficient decentralized collaborative LLM training, enabling multiple parties with limited GPU-memory and data resources to collectively train more capable LLMs than they could achieve individually. At the same time, this approach protects training data privacy of each participant by keeping training data, as well as parts of the forward pass signal and gradients locally within each party. By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances. Unlike most privacy-preserving schemes, which pay for confidentiality with lower task accuracy, our framework breaks that trade-off: across seven popular LLM benchmarks, it almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。