让专家与数据结构对齐，自动发现数据分组并优化专家网络。
Structure Aware Neural Architecture Search for Mixture of Experts
- 联合优化专家分配与各专家结构，显式建模数据聚类与专家匹配。
- 在无标签图像分类任务中95%聚类正确恢复，时间序列任务性能超越基线。
- 适合需要自动分组且无标签的复杂多领域任务，如跨域预测。
神经架构搜索（NAS）至今很少应用于混合专家（MoE）模型，现有设计将专家与数据结构的对齐留给模型自行学习。本文提出一种架构搜索框架，将数据聚类与专家分配的对齐作为显式搜索变量：同时优化数据聚类到专家的分配与各专家的网络结构。将联合问题建模为聚类感知的最大似然估计，证明其等价于潜在变量混合模型的不完备数据最大似然，并通过广义期望最大化（EM）算法求解；其中不可解析的专家质量项由自适应精炼的代理模型提供。证明了当代理误差可求和时，迭代过程收敛，且所有极限点均无法改进真实目标。在异构图像分类混合任务中，该方法在未见领域标签的情况下成功恢复95%的数据聚类；在该基准及一个四领域时间序列预测任务上，性能均优于使用无标签信息的MoE与NAS基线。
原文摘要 · Abstract (English)
Neural Architecture Search (NAS) has so far rarely been applied to Mixture-of-Experts (MoE) models, and existing MoE designs leave the alignment between experts and the structure of the data to emerge on its own. We propose an architecture search framework that makes this alignment an explicit search variable: the assignment of data clusters to experts is optimised jointly with the per-expert architectures. We cast the joint problem as a cluster-aware likelihood maximisation, show that it coincides with the incomplete-data maximum likelihood of a latent-variable mixture, and solve it by a generalised Expectation-Maximisation procedure whose otherwise intractable expert-quality term is supplied by an adaptively refined surrogate. We prove that the iterates converge whenever the surrogate errors are summable, and that at every limit point no candidate the search produces improves the true objective. On a heterogeneous image-classification mixture the method recovers the underlying domain partition on 95% of clusters without ever observing domain labels, and on that benchmark and a four-domain time-series forecasting one alike it outperforms the MoE and NAS baselines that likewise use no label information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。