用软最近邻损失让专家分工更明确，解决模型过拟合问题。
Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
- 先用软最近邻损失优化特征表示，使同类样本更接近
- 实验显示在四个数据集上分类准确率显著提升
- 适合研究专家分工、模型泛化能力的开发者参考
Mixture-of-Experts（MoE）模型通过门控网络调度多个专家网络对数据子集进行专精处理。但原始输入特征空间中类别边界重叠常导致专家塌陷——多个专家学习冗余表示，迫使门控网络采取僵化路由策略。本文提出一种增强型MoE架构：在门控与专家网络前引入基于软最近邻损失（SNNL）优化的特征提取器，预先将潜在空间中的同类样本距离最小化，从而缓解结构化专家塌陷，促使专家学习高度正交的权重。我们使用专家专精度熵和成对嵌入相似度量化该动态变化。在MNIST、FashionMNIST、CIFAR10和CIFAR100四个基准图像分类数据集上的实验表明，SNNL增强的MoE模型实现了结构差异化的专家，使门控网络可采用更灵活的路由策略。该范式显著提升了FashionMNIST、CIFAR10和CIFAR100上的分类准确率。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) model uses a set of expert networks that specialize on subsets of a dataset under the supervision of a gating network. A common issue in MoE architectures is ``expert collapse'' where overlapping class boundaries in the raw input feature space cause multiple experts to learn redundant representations, thus forcing the gating network into rigid routing to compensate. We propose an enhanced MoE architecture that utilizes a feature extractor network optimized using Soft Nearest Neighbor Loss (SNNL) prior to feeding input features to the gating and expert networks. By pre-conditioning the latent space to minimize distances among class-similar data points, we resolve structural expert collapse which results to experts learning highly orthogonal weights. We employ Expert Specialization Entropy and Pairwise Embedding Similarity to quantify this dynamic. We evaluate our experimental approach across four benchmark image classification datasets (MNIST, FashionMNIST, CIFAR10, and CIFAR100), and we show our SNNL-augmented MoE models demonstrate structurally diverse experts which allow the gating network to adopt a more flexible routing strategy. This paradigm significantly improves classification accuracy on the FashionMNIST, CIFAR10, and CIFAR100 datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。