通过特征空间主成分引导路由，解决专家模型负载不均与冗余问题。
EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts
- 基于学习的正交特征基进行输入投影和路由，实现几何分区。
- 无需辅助损失函数，同时缓解负载不均与专家同质化问题。
- 适合追求高效且多样化的专家模型设计者，尤其适用于大规模模型。
深度学习模型的持续扩展带来了难以承受的计算负担，使混合专家（MoE）架构成为提升效率的有力途径。然而，MoE模型面临两大核心挑战：一是‘富者愈富’现象导致的负载不均，少数专家被过度使用；二是专家同质化问题，即专家学习到相似表示，削弱了其分工意义。现有解决方案通常引入辅助负载均衡损失，虽缓解了不均，却常因强制均匀路由而加剧同质化。为此，我们提出基于特征基的混合专家（EMoE），其路由机制依赖于学习到的正交特征基。EMoE将输入标记投影至共享特征基，并根据其与特征空间主成分的对齐度进行路由。这种基于几何原理的数据划分方式，天然促进专家使用均衡与多样化专业能力发展，且无需冲突的辅助损失函数。代码已公开于 https://github.com/Belis0811/EMoE。
原文摘要 · Abstract (English)
The relentless scaling of deep learning models has led to unsustainable computational demands, positioning Mixture-of-Experts (MoE) architectures as a promising path towards greater efficiency. However, MoE models are plagued by two fundamental challenges: 1) a load imbalance problem known as the``rich get richer" phenomenon, where a few experts are over-utilized, and 2) an expert homogeneity problem, where experts learn redundant representations, negating their purpose. Current solutions typically employ an auxiliary load-balancing loss that, while mitigating imbalance, often exacerbates homogeneity by enforcing uniform routing at the expense of specialization. To resolve this, we introduce the Eigen-Mixture-of-Experts (EMoE), a novel architecture that leverages a routing mechanism based on a learned orthonormal eigenbasis. EMoE projects input tokens onto this shared eigenbasis and routes them based on their alignment with the principal components of the feature space. This principled, geometric partitioning of data intrinsically promotes both balanced expert utilization and the development of diverse, specialized experts, all without the need for a conflicting auxiliary loss function. Our code is publicly available at https://github.com/Belis0811/EMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。