探究专家模型路由对上下文的敏感性，发现编码层更依赖语义关联。
Routing in Sparsely-gated Language Models responds to Context
- 通过相似文本对追踪路由决策，分析上下文影响
- 编码层路由主要由语义关联决定，上下文起微调作用
- 解码层路由变化大且对上下文不敏感，适合特定场景研究
语言模型近年采用混合专家结构（由路由器和多个专家组成），在固定计算预算下提升参数量。基于先前研究指出，词元-专家分配主要受词元身份和位置影响，本文通过追踪语义标注的文本对路由决策，评估学习到的词元-专家分配对上下文的敏感性。结果发现：编码层路由主要依赖（语义）关联，而上下文线索提供额外精炼；相反，解码层路由变化更大，对上下文显著不敏感。
原文摘要 · Abstract (English)
Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computational budget. Building on previous efforts indicating that token-expert assignments are predominantly influenced by token identities and positions, we trace routing decisions of similarity-annotated text pairs to evaluate the context sensitivity of learned token-expert assignments. We observe that routing in encoder layers mainly depends on (semantic) associations, but contextual cues provide an additional layer of refinement. Conversely, routing in decoder layers is more variable and markedly less sensitive to context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。