通过局部差分隐私约束门控机制，显著收紧专家混合模型的风险上界。
Tighter Risk Bounds for Mixtures of Experts
- 对一选一门控机制施加局部差分隐私，改进风险上界。
- 风险上界对专家数量呈对数依赖，优于现有方法。
- 实验验证该方法提升泛化能力，且隐私约束可行。
本文针对采用一选一门控机制的专家混合模型，通过在门控机制上施加局部差分隐私（LDP），给出了风险的上界。与传统的全选门控机制不同，该方法使风险上界对专家数量仅呈对数依赖，并将门控机制的影响封装于LDP参数中，在合理条件下显著优于已有边界。实验结果支持理论分析，表明该方法提升了专家混合模型的泛化性能，验证了在门控机制上施加LDP的可行性。
原文摘要 · Abstract (English)
In this work, we provide upper bounds on the risk of mixtures of experts by imposing local differential privacy (LDP) on their gating mechanism. These theoretical guarantees are tailored to mixtures of experts that utilize the one-out-of-$n$ gating mechanism, as opposed to the conventional $n$-out-of-$n$ mechanism. The bounds exhibit logarithmic dependence on the number of experts, and encapsulate the dependence on the gating mechanism in the LDP parameter, making them significantly tighter than existing bounds, under reasonable conditions. Experimental results support our theory, demonstrating that our approach enhances the generalization ability of mixtures of experts and validating the feasibility of imposing LDP on the gating mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。