用公开数据代理私有数据，让分散训练的专家模型安全融合。
MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
- 通过多样性感知选择公开数据中的代表性样本作为代理
- 在多个视觉与语言任务上超越现有隐私保护融合方法
- 适合需要保护数据隐私的分布式专家模型整合场景
Mixture-of-Experts（MoE）模型通过组合专用专家来扩展容量，但现有方法多假设可集中访问训练数据。现实中数据分布于各客户端，因隐私限制无法共享，导致统一训练困难。我们提出MetaMoE，一种隐私保护框架，利用公开代理数据作为不可访问私有数据的替代，将独立训练、领域专精的专家统一为单一MoE。MetaMoE的核心是多样性感知的代理选择，从公开数据中选取与客户端域相关且多样化的样本，有效逼近私有数据分布并监督路由器学习。这些代理还用于对齐专家训练，提升统一时的专家协作能力，同时引入上下文感知路由器以应对异构输入。在计算机视觉和自然语言处理基准上的实验表明，MetaMoE持续优于近期隐私保护的MoE统一方法。代码已开源：https://github.com/ws-jiang/MetaMoE。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE, a privacy-preserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain-relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ws-jiang/MetaMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。