研究专家混合模型中路由机制如何根据词性分配文本，发现专家有明确分工。
Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models
- 通过词性标签分析专家路由路径，揭示专家对特定词性的偏好。
- 六种主流模型均显示专家对名词、动词等词性有显著处理专长。
- 适合对模型内部机制和自然语言处理优化感兴趣的读者。
本研究考察了混合专家（MoE）模型中模型内嵌路由机制的行为，重点关注标记依据其语言特征（特别是词性，POS）进行路由的方式。目标是探究不同MoE架构下，专家是否专门处理具有相似语言特性的标记。通过分析标记在各专家和层间的流动轨迹，旨在揭示MoE模型如何处理语言信息。对六种流行MoE模型的分析发现，专家对特定词性类别表现出显著的专业化，路由路径对词性具有高度预测准确性，凸显了路由路径在表征标记方面的价值。
原文摘要 · Abstract (English)
This study investigates the behavior of model-integrated routers in Mixture of Experts (MoE) models, focusing on how tokens are routed based on their linguistic features, specifically Part-of-Speech (POS) tags. The goal is to explore across different MoE architectures whether experts specialize in processing tokens with similar linguistic traits. By analyzing token trajectories across experts and layers, we aim to uncover how MoE models handle linguistic information. Findings from six popular MoE models reveal expert specialization for specific POS categories, with routing paths showing high predictive accuracy for POS, highlighting the value of routing paths in characterizing tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。