MoE模型能自动发现数据隐含聚类结构，而普通网络做不到。
Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
- 用路由机制将输入分发给专用专家，分解复杂回归任务
- 实验表明MoE在单索引模型聚类结构下可成功学习，普通网络失败
- 首次从SGD动态角度证明MoE在非线性回归中的理论优势
Mixture of Experts（MoE）是一种由多个专用模型组成的集成架构，通过路由机制将每个输入分配给合适的专家，已在机器学习领域取得成功。然而，由于其内在复杂性，该架构的理论理解滞后。本文首次从理论上研究了在随机梯度下降（SGD）下学习具有单索引模型聚类结构的回归任务时，MoE的样本复杂度和运行时间复杂度。一方面，我们证明普通神经网络无法检测这种潜在组织结构，因为它只能将问题整体处理，这与信息指数较低但整体上升的特性有关。另一方面，我们展示了MoE能够利用每个专家弱恢复单一聚类对应简单函数的能力，将问题分解为更易处理的子问题。这是首个通过分析非线性回归中MoE的SGD动态来揭示其优势的研究。
原文摘要 · Abstract (English)
Mixture of Experts (MoE), an ensemble of specialized models equipped with a router that dynamically distributes each input to appropriate experts, has achieved successful results in the field of machine learning. However, theoretical understanding of this architecture is falling behind due to its inherent complexity. In this paper, we theoretically study the sample and runtime complexity of MoE following the stochastic gradient descent (SGD) when learning a regression task with an underlying cluster structure of single index models. On the one hand, we prove that a vanilla neural network fails in detecting such a latent organization as it can only process the problem as a whole. This is intrinsically related to the concept of information exponent which is low for each cluster, but increases when we consider the entire task. On the other hand, we show that a MoE succeeds in dividing this problem into easier subproblems by leveraging the ability of each expert to weakly recover the simpler function corresponding to an individual cluster. To the best of our knowledge, this work is among the first to explore the benefits of the MoE framework by examining its SGD dynamics in the context of nonlinear regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。