发现稀疏专家模型的路由有任务特征,可用来识别输入任务类型。
Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
- 用路由签名捕捉专家激活模式,分析任务相关性。
- 同类任务路由相似度达0.84,跨类仅0.62,差异显著。
- 仅凭路由签名就能92.5%准确分类任务,适合模型可解释性研究。
稀疏混合专家(MoE)架构通过条件计算实现大语言模型高效扩展,但其路由机制仍不明确。本文提出路由签名,一种总结特定提示在各层专家激活模式的向量表示,并用于检验路由是否具有任务依赖结构。以OLMoE-1B-7B-0125-Instruct为实验平台,发现同类别提示产生高度相似的路由签名(0.8435 ± 0.0879),而不同类别间相似度显著更低(0.6225 ± 0.1687),对应Cohen's d = 1.44。仅基于路由签名的逻辑回归分类器在四分类任务上达到92.5% ± 6.1%的交叉验证准确率。通过置换和负载均衡基线验证,结果非由稀疏性或平衡约束单独导致。进一步分析显示,任务结构在深层逐渐显现。结果表明,路由不仅是负载均衡工具,更是可测量的任务敏感组件。论文发布MOE-XRAY工具包,用于路由监控与分析。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。