通过双阶段聚类提升MoE模型在图像分类中的表现。
Double-Stage Feature-Level Clustering-Based Mixture of Experts Framework
- 先对输入特征聚类，再用伪标签标注大量无标签数据。
- 在三个基准数据集上实现与密集模型相当的分类精度。
- 适合处理含噪声和异常值的数据，尤其适用于小样本标注场景。
Mixture-of-Experts（MoE）模型在深度学习中表现优异，但其复杂架构及其相较于密集模型在图像分类中的优势仍不明确。以往研究中，输入空间的噪声和异常值常影响MoE性能。虽有方法引入输入聚类训练MoE，但多数聚类算法无法利用标签数据，限制了效果。本文提出双阶段特征级聚类与伪标签驱动的MoE框架（DFCP-MoE），包含输入特征提取、特征级聚类及高效的伪标签策略。该方法在减少噪声与异常值影响的同时，利用少量标注数据为大量无标签输入打标。我们设计了一种条件端到端联合训练机制，通过在已良好标注的聚类输入上训练MoE模型，提升专家专业化程度。与传统MoE和密集模型不同，DFCP-MoE能有效捕捉输入空间多样性，实现具有竞争力的推理结果。我们在三个多分类任务的基准数据集上验证了该方法的有效性。
原文摘要 · Abstract (English)
The Mixture-of-Experts (MoE) model has succeeded in deep learning (DL). However, its complex architecture and advantages over dense models in image classification remain unclear. In previous studies, MoE performance has often been affected by noise and outliers in the input space. Some approaches incorporate input clustering for training MoE models, but most clustering algorithms lack access to labeled data, limiting their effectiveness. This paper introduces the Double-stage Feature-level Clustering and Pseudo-labeling-based Mixture of Experts (DFCP-MoE) framework, which consists of input feature extraction, feature-level clustering, and a computationally efficient pseudo-labeling strategy. This approach reduces the impact of noise and outliers while leveraging a small subset of labeled data to label a large portion of unlabeled inputs. We propose a conditional end-to-end joint training method that improves expert specialization by training the MoE model on well-labeled, clustered inputs. Unlike traditional MoE and dense models, the DFCP-MoE framework effectively captures input space diversity, leading to competitive inference results. We validate our approach on three benchmark datasets for multi-class classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。