揭示视觉专家模型中专家的真实编码机制，超越路由表象。
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
- 用对比学习训练稀疏门控卷积专家模型，从专家层面分析其特化能力。
- 专家对动/非动物体的区分稳定且贯穿路由到输出全过程。
- 专家在连续视觉与语义维度上具广泛调谐，适合研究模型内部表征者。
Mixture-of-Experts (MoE) 模型常通过分析类别路由至哪个专家来解释。然而，仅靠路由无法揭示每个专家实际编码内容。我们基于自然图像，采用对比学习训练稀疏门控卷积 MoE 模型，并运用视觉神经科学工具刻画专家特化。从门控层扩展到专家层，测量每专家的类别可分性及对最显著输入的调谐程度；从类别层扩展到特征层，通过人类行为判断数据集（THINGS）推导语义维度解释调谐机制。最后，利用调谐与表征相似性分析评估不同初始化下专家分配的稳定性。结果表明，动/非动物区分主导专家划分，从门控到读出均明显，且跨独立训练模型稳定。尽管路由显示稀疏的类别偏好，专家分析揭示其对连续视觉与语义维度更广泛的调谐，超越类别边界。各专家虽特征调谐不同，但类别可分性相近，说明超越类别分析具有解释优势。这些结果表明，视觉 MoE 中的专家特化远超类别路由，需通过精细的专家级调谐与表征结构探究。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。