无监督路由让专家模型发现超越人类分类的隐藏结构
Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
- 用无监督方式让专家自动分工，不依赖人工标签
- 在QuickDraw数据集上重建效果优于有监督基线
- 适合研究模型内在结构与专家分工机制的人
理解神经网络内部组织仍是深度学习可解释性的核心挑战。本文提出一种新型稀疏专家混合变分自编码器（SMoE-VAE），在QuickDraw数据集上对比了无监督专家路由与有监督基线（基于真实标签）的表现。令人惊讶的是，无监督路由始终取得更优的重构性能。专家自发学习到有意义的子类别结构，这些结构常超出人为定义的类别边界。通过t-SNE可视化和重构分析，我们发现MoE模型揭示的数据底层结构更符合模型目标，而非预设标签。此外，对数据集规模影响的研究揭示了数据量与专家专业化之间的权衡，为高效MoE架构设计提供指导。
原文摘要 · Abstract (English)
Understanding the internal organization of neural networks remains a fundamental challenge in deep learning interpretability. We address this challenge by exploring a novel Sparse Mixture of Experts Variational Autoencoder (SMoE-VAE) architecture. We test our model on the QuickDraw dataset, comparing unsupervised expert routing against a supervised baseline guided by ground-truth labels. Surprisingly, we find that unsupervised routing consistently achieves superior reconstruction performance. The experts learn to identify meaningful sub-categorical structures that often transcend human-defined class boundaries. Through t-SNE visualizations and reconstruction analysis, we investigate how MoE models uncover fundamental data structures that are more aligned with the model's objective than predefined labels. Furthermore, our study on the impact of dataset size provides insights into the trade-offs between data quantity and expert specialization, offering guidance for designing efficient MoE architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。